System Call Management in a User-Mode, Multi-Threaded, Self-Scheduling Processor

ABSTRACT

Representative apparatus, method, and system embodiments are disclosed for a self-scheduling processor which also provides additional functionality. Representative embodiments include a self-scheduling processor, comprising: a processor core adapted to execute a received instruction; and a core control circuit adapted to automatically schedule an instruction for execution by the processor core in response to a received work descriptor data packet. In another embodiment, the core control circuit is also adapted to schedule a fiber create instruction for execution by the processor core, to reserve a predetermined amount of memory space in a thread control memory to store return arguments, and to generate one or more work descriptor data packets to another processor or hybrid threading fabric circuit for execution of a corresponding plurality of execution threads. Event processing, data path management, system calls, memory requests, and other new instructions are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of andpriority to U.S. patent application Ser. No. 16/399,702, filed Apr. 30,2019, inventor Tony M. Brewer, titled “System Call Management in aUser-Mode, Multi-Threaded, Self-Scheduling Processor”, which is anonprovisional of and claims the benefit of and priority to U.S.Provisional Patent Application No. 62/667,717, filed May 7, 2018,inventor Tony M. Brewer, titled “System Call Management in a User-Mode,Multi-Threaded, Self-Scheduling Processor”, which are commonly assignedherewith, and all of which are hereby incorporated herein by referencein their entireties with the same full force and effect as if set forthin their entireties herein.

FIELD OF THE INVENTION

The present invention, in general, relates to configurable computingcircuitry, and more particularly, relates to a heterogeneous computingsystem which includes a self-scheduling processor, configurablecomputing circuitry with an embedded interconnection network, dynamicreconfiguration, and dynamic control over energy or power consumption.

BACKGROUND OF THE INVENTION

Many existing computing systems have reached significant limits forcomputation processing capabilities, both in terms of speed ofcomputation, energy (or power) consumption, and associated heatdissipation. For example, existing computing solutions have becomeincreasingly inadequate as the need for advanced computing technologiesgrows, such as to accommodate artificial intelligence and othersignificant computing applications.

Accordingly, there is an ongoing need for a computing architecturecapable of providing high performance using sparse data sets, involvinglimited or no data reuse, which typically cause poor cache hit rates.Such a computing architecture should be tolerant of latency to memoryand allow high sustained executed instructions per clock.

There is also an ongoing need for a computing architecture capable ofproviding high performance and energy efficient solutions forcompute-intensive kernels, such as for computation of Fast FourierTransforms (FFTs) and finite impulse response (FIR) filters used insensing, communication, and analytic applications, such as syntheticaperture radar, 5G base stations, and graph analytic applications suchas graph clustering using spectral techniques, machine learning, 5Gnetworking algorithms, and large stencil codes, for example and withoutlimitation.

There is also an ongoing need for a processor architecture capable ofsignificant parallel processing and further interacting with andcontrolling a configurable computing architecture for performance of anyof these various applications.

SUMMARY OF THE INVENTION

As discussed in greater detail below, the representative apparatus,system and method provide for a computing architecture capable ofproviding high performance and energy efficient solutions forcompute-intensive kernels, such as for computation of Fast FourierTransforms (FFTs) and finite impulse response (FIR) filters used insensing, communication, and analytic applications, such as syntheticaperture radar, 5G base stations, and graph analytic applications suchas graph clustering using spectral techniques, machine learning, 5Gnetworking algorithms, and large stencil codes, for example and withoutlimitation.

As mentioned above, sparse data sets typically cause poor cache hitrates. The representative apparatus, system and method provide for acomputing architecture capable of allowing some threads to be waitingfor response from memory while other threads are continuing to executeinstructions. This style of compute is tolerant of latency to memory andallows high sustained executed instructions per clock.

Also as discussed in greater detail below, the representative apparatus,system and method provide for a processor architecture capable ofself-scheduling, significant parallel processing and further interactingwith and controlling a configurable computing architecture forperformance of any of these various applications.

A self-scheduling processor is disclosed. In a representativeembodiment, the processor comprises: a processor core adapted to executea received instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule an instruction for execution by the processor core in responseto a received work descriptor data packet. In another representativeembodiment, the processor comprises: a processor core adapted to executea received instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule an instruction for execution by the processor core in responseto a received event data packet.

A multi-threaded, self-scheduling processor is also disclosed which cancreate threads on local or remote compute elements. In a representativeembodiment, the processor comprises: a processor core adapted to executea fiber create instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule the fiber create instruction for execution by the processorcore and generate one or more work descriptor data packets to anotherprocessor or hybrid threading fabric circuit for execution of acorresponding plurality of execution threads. In another representativeembodiment, the processor comprises: a processor core adapted to executea fiber create instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to schedule the fibercreate instruction for execution by the processor core, to reserve apredetermined amount of memory space in a thread control memory to storereturn arguments, and to generate one or more work descriptor datapackets to another processor or hybrid threading fabric circuit forexecution of a corresponding plurality of execution threads.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interface; athread control memory coupled to the interconnection network interface;an execution queue coupled to the thread control memory; a control logicand thread selection circuit coupled to the execution queue, to thethread control memory; and an instruction cache coupled to the controllogic and thread selection circuit; and further, a processor core iscoupled to the instruction cache of the core control circuit.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interface; athread control memory coupled to the interconnection network interface;a network response memory; an execution queue coupled to the threadcontrol memory; a control logic and thread selection circuit coupled tothe execution queue, to the thread control memory; an instruction cachecoupled to the control logic and thread selection circuit; and a commandqueue; and further, a processor core is coupled to the instruction cacheand to the command queue of the core control circuit.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; an execution queue coupled to the thread control memory; and acontrol logic and thread selection circuit coupled to the executionqueue, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to automaticallyplace the thread identifier in the execution queue, and to periodicallyselect the thread identifier for execution of the execution thread.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; an execution queue coupled to the thread control memory; and acontrol logic and thread selection circuit coupled to the executionqueue, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to automaticallyplace the thread identifier in the execution queue, and to periodicallyselect the thread identifier for execution of an instruction of anexecution thread by a processor core.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an execution queue coupled to thethread control memory; and a control logic and thread selection circuitcoupled to the execution queue, the control logic and thread selectioncircuit adapted to assign an available thread identifier to theexecution thread, to automatically place the thread identifier in theexecution queue, and to periodically select the thread identifier forexecution of an instruction of an execution thread by the processorcore.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, a data cache,and a general purpose register storing a received argument; an executionqueue coupled to the thread control memory; and a control logic andthread selection circuit coupled to the execution queue, the controllogic and thread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue, and to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core, the processor core using data stored in the datacache or general purpose register.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, and threadstate registers storing a valid state or a paused state for each threadidentifier of the plurality of thread identifiers; an execution queuecoupled to the thread control memory; and a control logic and threadselection circuit coupled to the execution queue, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue when it has a valid state, and for aslong as the valid state remains, to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core until completion of the execution thread.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, and threadstate registers storing a valid state or a paused state for each threadidentifier of the plurality of thread identifiers; an execution queuecoupled to the thread control memory; and a control logic and threadselection circuit coupled to the execution queue, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue when it has a valid state, and for aslong as the valid state remains, to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core, and to pause thread execution by not returning thethread identifier to the execution queue when it has a pause state.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, athread state register, a program count register storing a receivedprogram count, a data cache, and a general purpose register storing areceived argument; an execution queue coupled to the thread controlmemory; and a control logic and thread selection circuit coupled to theexecution queue, the control logic and thread selection circuit adaptedto assign an available thread identifier to the execution thread, toautomatically place the thread identifier in the execution queue, and toperiodically select the thread identifier for execution of aninstruction of an execution thread by the processor core.

In another representative embodiment, a processor comprises: a processorcore adapted to execute a plurality of instructions; and a core controlcircuit coupled to the processor core, with the core control circuitcomprising: an interconnection network interface coupleable to aninterconnection network to receive a work descriptor data packet, todecode the received work descriptor data packet into an execution threadhaving an initial program count and any received argument; a threadcontrol memory coupled to the interconnection network interface andcomprising a plurality of registers, the plurality of registerscomprising a thread identifier pool register storing a plurality ofthread identifiers, a thread state register, a program count registerstoring the received program count, a data cache, and a general purposeregister storing the received argument; an execution queue coupled tothe thread control memory; a control logic and thread selection circuitcoupled to the execution queue and to the thread control memory, thecontrol logic and thread selection circuit adapted to assign anavailable thread identifier to the execution thread, to place the threadidentifier in the execution queue, to select the thread identifier forexecution, to access the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; and an instruction cache coupled to the processor coreand to the control logic and thread selection circuit to receive theinitial program count and provide to the processor core a correspondinginstruction for execution, of the plurality of instructions.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; a thread control memory coupled to the interconnection networkinterface and comprising a plurality of registers, the plurality ofregisters comprising a thread identifier pool register storing aplurality of thread identifiers, a thread state register, a programcount register storing the received program count, a data cache, and ageneral purpose register storing the received argument; an executionqueue coupled to the thread control memory; a control logic and threadselection circuit coupled to the execution queue and to the threadcontrol memory, the control logic and thread selection circuit adaptedto assign an available thread identifier to the execution thread, toautomatically place the thread identifier in the execution queue, toperiodically select the thread identifier for execution, to access thethread control memory using the thread identifier as an index to selectthe initial program count for the execution thread; and an instructioncache coupled to the control logic and thread selection circuit toreceive the initial program count and provide a correspondinginstruction for execution; and further, a processor core is coupled tothe instruction cache of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; a thread control memory coupled to the interconnection networkinterface and comprising a plurality of registers, the plurality ofregisters comprising a thread identifier pool register storing aplurality of thread identifiers, a thread state register, a programcount register storing the received program count, and a general purposeregister storing the received argument; an execution queue coupled tothe thread control memory; a control logic and thread selection circuitcoupled to the execution queue and to the thread control memory, thecontrol logic and thread selection circuit adapted to assign anavailable thread identifier to the execution thread, to place the threadidentifier in the execution queue, to select the thread identifier forexecution, to access the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; an instruction cache coupled to the control logic andthread selection circuit to receive the initial program count andprovide a corresponding instruction for execution; and a command queue;and further, a processor core is coupled to the instruction cache andthe command queue of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit coupled to the interconnection network interface andcomprising: an interconnection network interface coupleable to aninterconnection network to receive a work descriptor data packet, todecode the received work descriptor data packet into an execution threadhaving an initial program count and any received argument; a threadcontrol memory coupled to the interconnection network interface andcomprising a plurality of registers, the plurality of registerscomprising a thread identifier pool register storing a plurality ofthread identifiers, a thread state register, a program count registerstoring the received program count, and a general purpose registerstoring the received argument; an execution queue coupled to the threadcontrol memory; a control logic and thread selection circuit coupled tothe execution queue and to the thread control memory, the control logicand thread selection circuit adapted to assign an available threadidentifier to the execution thread, to place the thread identifier inthe execution queue, to select the thread identifier for execution, toaccess the thread control memory using the thread identifier as an indexto select the initial program count for the execution thread, and aninstruction cache coupled to the control logic and thread selectioncircuit to receive the initial program count and provide a correspondinginstruction for execution; and further, a processor core is coupled tothe instruction cache of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a call workdescriptor data packet, to decode the received work descriptor datapacket into an execution thread having an initial program count and anyreceived argument, and to encode a work descriptor packet fortransmission to other processing elements; a thread control memorycoupled to the interconnection network interface and comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, athread state register, a program count register storing the receivedprogram count, and a general purpose register storing the receivedargument; an execution queue coupled to the thread control memory; anetwork response memory coupled to the interconnection networkinterface; a control logic and thread selection circuit coupled to theexecution queue, to the thread control memory, and to the instructioncache, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to place thethread identifier in the execution queue, to select the threadidentifier for execution, to access the thread control memory using thethread identifier as an index to select the initial program count forthe execution thread; an instruction cache coupled to the control logicand thread selection circuit to receive the initial program count andprovide a corresponding instruction for execution; and a command queuestoring one or more commands for generation of one or more workdescriptor packets; and further, a processor core is coupled to theinstruction cache and the command queue of the core control circuit, theprocessor core adapted to execute the corresponding instruction.

For any of the various representative embodiments, the core controlcircuit may further comprise: an interconnection network interfacecoupleable to an interconnection network, the interconnection networkinterface adapted to receive a work descriptor data packet, to decodethe received work descriptor data packet into an execution thread havingan initial program count and any received argument. For any of thevarious representative embodiments, the interconnection networkinterface may be further adapted to receive an event data packet, todecode the received event data packet into an event identifier and anyreceived argument.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the interconnection network interface, the controllogic and thread selection circuit adapted to assign an available threadidentifier to the execution thread.

For any of the various representative embodiments, the core controlcircuit may further comprise: a thread control memory having a pluralityof registers, with the plurality of registers comprising one or more ofthe following, in any selected combination: a thread identifier poolregister storing a plurality of thread identifiers; a thread stateregister; a program count register storing a received initial programcount; a general purpose register storing the received argument; apending fiber return count register; a return argument buffer orregister; a return argument link list register; a custom atomictransaction identifier register; an event state register; an event maskregister; and a data cache.

For any of the various representative embodiments, the interconnectionnetwork interface may be further adapted to store the execution threadhaving the initial program count and any received argument in the threadcontrol memory using a thread identifier as an index to the threadcontrol memory.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the thread control memory and to the interconnectionnetwork interface, the control logic and thread selection circuitadapted to assign an available thread identifier to the executionthread.

For any of the various representative embodiments, the core controlcircuit may further comprise: an execution queue coupled to the threadcontrol memory, the execution queue storing one or more threadidentifiers.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the execution queue, to the interconnection networkinterface, and to the thread control memory, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to place the thread identifier inthe execution queue, to select the thread identifier for execution, andto access the thread control memory using the thread identifier as anindex to select the initial program count for the execution thread.

For any of the various representative embodiments, the core controlcircuit may further comprise: an instruction cache coupled to thecontrol logic and thread selection circuit to receive the initialprogram count and provide a corresponding instruction for execution.

In another representative embodiment, the processor further may furthercomprise: a processor core coupled to the instruction cache of the corecontrol circuit, the processor core adapted to execute the correspondinginstruction.

For any of the various representative embodiments, the core controlcircuit may be further adapted to assign an initial valid state to theexecution thread. For any of the various representative embodiments, thecore control circuit may be further adapted to assign a pause state tothe execution thread in response to the processor core executing amemory load instruction. For any of the various representativeembodiments, the core control circuit may be further adapted to assign apause state to the execution thread in response to the processor coreexecuting a memory store instruction.

For any of the various representative embodiments, the core controlcircuit may be further adapted to end execution of a selected thread inresponse to the execution of a return instruction by the processor core.For any of the various representative embodiments, the core controlcircuit may be further adapted to return a corresponding threadidentifier of the selected thread to the thread identifier pool registerin response to the execution of a return instruction by the processorcore. For any of the various representative embodiments, the corecontrol circuit may be further adapted to clear the registers of thethread control memory indexed by the corresponding thread identifier ofthe selected thread in response to the execution of a return instructionby the processor core.

For any of the various representative embodiments, the interconnectionnetwork interface may be further adapted to generate a return workdescriptor packet in response to the execution of a return instructionby the processor core.

For any of the various representative embodiments, the core controlcircuit may further comprise: a network response memory. For any of thevarious representative embodiments, the network response memory maycomprise one or more of the following, in any selected combination: amemory request register; a thread identifier and transaction identifierregister; a request cache line index register; a bytes register; and ageneral purpose register index and type register.

For any of the various representative embodiments, the interconnectionnetwork interface may be adapted to generate a point-to-point event datamessage. For any of the various representative embodiments, theinterconnection network interface may be adapted to generate a broadcastevent data message.

For any of the various representative embodiments, the core controlcircuit may be further adapted to use an event mask stored in the eventmask register to respond to a received event data packet. For any of thevarious representative embodiments, the core control circuit may befurther adapted to determine an event number corresponding to a receivedevent data packet. For any of the various representative embodiments,the core control circuit may be further adapted to change the status ofa thread identifier from pause to valid in response to a received eventdata packet to resume execution of a corresponding execution thread. Forany of the various representative embodiments, the core control circuitmay be further adapted to change the status of a thread identifier frompause to valid in response to an event number of a received event datapacket to resume execution of a corresponding execution thread.

For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to successively select anext thread identifier from the execution queue for execution of asingle instruction of a corresponding execution thread. For any of thevarious representative embodiments, the control logic and threadselection circuit may be further adapted to perform a round-robinselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread. For any of the variousrepresentative embodiments, the control logic and thread selectioncircuit may be further adapted to perform a round-robin selection of anext thread identifier from the execution queue, of the plurality ofthread identifiers, each for execution of a single instruction of acorresponding execution thread until completion of the execution thread.For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to perform a barrelselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread.

For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to assign a valid statusor a pause status to a thread identifier. For any of the variousrepresentative embodiments, the control logic and thread selectioncircuit may be further adapted to assign a priority status to a threadidentifier. For any of the various representative embodiments, thecontrol logic and thread selection circuit may be further adapted to,following execution of a corresponding instruction, to return thecorresponding thread identifier to the execution queue with an assignedvalid status and an assigned priority.

For any of the various representative embodiments, the core controlcircuit may further comprise: a network command queue coupled to theprocessor core.

For any of the various representative embodiments, the interconnectionnetwork interface may comprise: an input queue; a packet decoder circuitcoupled to the input queue, to the control logic and thread selectioncircuit, and to the thread control memory; an output queue; and a packetencoder circuit coupled to the output queue, to the network responsememory, and to the network command queue.

For any of the various representative embodiments, the execution queuemay further comprise: a first priority queue; and a second priorityqueue. For any of the various representative embodiments, the controllogic and thread selection circuit may further comprise: threadselection control circuitry coupled to the execution queue, the threadselection control circuitry adapted to select a thread identifier fromthe first priority queue at a first frequency and to select a threadidentifier from the second priority queue at a second frequency, thesecond frequency lower than the first frequency. For any of the variousrepresentative embodiments, the thread selection control circuitry maybe adapted to determine the second frequency as a skip count fromselection of a thread identifier from the first priority queue.

For any of the various representative embodiments, the core controlcircuit may further comprise: data path control circuitry adapted tocontrol access size over the first interconnection network. For any ofthe various representative embodiments, the core control circuit mayfurther comprise: data path control circuitry adapted to increase ordecrease memory load access size in response to time averaged usagelevels. For any of the various representative embodiments, the corecontrol circuit may further comprise: data path control circuitryadapted to increase or decrease memory store access size in response totime averaged usage levels. For any of the various representativeembodiments, the control logic and thread selection circuit may befurther adapted to increase a size of a memory load access request tocorrespond to a cache line boundary of the data cache.

For any of the various representative embodiments, the core controlcircuit may further comprise: system call circuitry adapted to generateone or more system calls to a host processor. For any of the variousrepresentative embodiments, the system call circuitry may furthercomprise: a plurality of system call credit registers storing apredetermined credit count to modulate a number of system calls in anypredetermined period of time.

For any of the various representative embodiments, the core controlcircuit may be further adapted, in response to a request from a hostprocessor, to generate a command to the command queue for theinterconnection network interface to copy and transmit all data from thethread control memory corresponding to a selected thread identifier formonitoring thread state.

For any of the various representative embodiments, the processor coremay be adapted to execute a fiber create instruction to generate one ormore commands to the command queue for the interconnection networkinterface to generate one or more call work descriptor packets toanother processor core or to a hybrid threading fabric circuit. For anyof the various representative embodiments, the core control circuit maybe further adapted, in response to execution of a fiber createinstruction by the processor core, to reserve a predetermined amount ofmemory space in the general purpose registers or return argumentregisters.

For any of the various representative embodiments, in response to thegeneration of one or more call work descriptor packets to anotherprocessor core or to a hybrid threading fabric, the core control circuitmay be adapted to store a thread return count in the thread returnregister. For any of the various representative embodiments, in responseto receipt of a return data packet, the core control circuit may beadapted to decrement the thread return count stored in the thread returnregister. For any of the various representative embodiments, in responseto the thread return count in the thread return register beingdecremented to zero, the core control circuit may be adapted to change apaused status to a valid status for a corresponding thread identifierfor subsequent execution of a thread return instruction for completionof the created fibers or threads.

For any of the various representative embodiments, the processor coremay be adapted to execute a waiting or nonwaiting fiber joininstruction. For any of the various representative embodiments, theprocessor core may be adapted to execute a fiber join all instruction.

For any of the various representative embodiments, the processor coremay be adapted to execute a non-cached read or load instruction todesignate a general purpose register for storage of data received from amemory. For any of the various representative embodiments, the processorcore may be adapted to execute a non-cached write or store instructionto designate data in a general purpose register for storage in a memory.

For any of the various representative embodiments, the core controlcircuit may be adapted to assign a transaction identifier to any load orstore request to memory and to correlate the transaction identifier witha thread identifier.

For any of the various representative embodiments, the processor coremay be adapted to execute a first thread priority instruction to assigna first priority to an execution thread having a corresponding threadidentifier. For any of the various representative embodiments, theprocessor core may be adapted to execute a second thread priorityinstruction to assign a second priority to an execution thread having acorresponding thread identifier.

For any of the various representative embodiments, the processor coremay be adapted to execute a custom atomic return instruction to completean executing thread of a custom atomic operation. For any of the variousrepresentative embodiments, in conjunction with a memory controller, theprocessor core may be adapted to execute a floating point atomic memoryoperation. For any of the various representative embodiments, inconjunction with a memory controller, the processor core may be adaptedto execute a custom atomic memory operation.

A method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; and automatically scheduling theinstruction for execution in response to the received work descriptordata packet.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivingan event data packet; and automatically scheduling the instruction forexecution in response to the received event data packet.

A method of a first processing element to generate a plurality ofexecution threads for performance by a second processing element is alsodisclosed, with a representative method embodiment comprising: executinga fiber create instruction; and in response to the execution of thefiber create instruction generating one or more work descriptor datapackets to the second processing element for execution of the pluralityof execution threads.

A method of a first processing element to generate a plurality ofexecution threads for performance by a second processing element is alsodisclosed, with a representative method embodiment comprising: executinga fiber create instruction; and in response to the execution of thefiber create instruction reserving a predetermined amount of memoryspace in a thread control memory to store return arguments andgenerating one or more work descriptor data packets to the secondprocessing element for execution of the plurality of execution threads.

A method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier forexecution of the execution thread; and periodically selecting the threadidentifier for execution of the execution thread.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier forexecution of the execution thread when it has a valid state; and for aslong as the valid state remains, periodically selecting the threadidentifier for execution of an instruction of the execution thread untilcompletion of the execution thread.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier in anexecution queue for execution of the execution thread when it has avalid state; and for as long as the valid state remains, periodicallyselecting the thread identifier for execution of an instruction of theexecution thread; and pausing thread execution by not returning thethread identifier to the execution queue when it has a pause state.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; storing the initial program count and anyreceived argument in a thread control memory; assigning an availablethread identifier to the execution thread; automatically queuing thethread identifier for execution of the execution thread when it has avalid state;

-   accessing the thread control memory using the thread identifier as    an index to select the initial program count for the execution    thread; and for as long as the valid state remains, periodically    selecting the thread identifier for execution of an instruction of    the execution thread until completion of the execution thread.

For any of the various representative embodiments, the method mayfurther comprise: receiving an event data packet; and decoding thereceived event data packet into an event identifier and any receivedargument.

For any of the various representative embodiments, the method mayfurther comprise: assigning an initial valid state to the executionthread.

For any of the various representative embodiments, the method mayfurther comprise: assigning a pause state to the execution thread inresponse to the execution of a memory load instruction. For any of thevarious representative embodiments, the method may further comprise:assigning a pause state to the execution thread in response to theexecution of a memory store instruction.

For any of the various representative embodiments, the method mayfurther comprise: terminating execution of a selected thread in responseto the execution of a return instruction. For any of the variousrepresentative embodiments, the method may further comprise: returning acorresponding thread identifier of the selected thread to the threadidentifier pool in response to the execution of a return instruction.For any of the various representative embodiments, the method mayfurther comprise: clearing the registers of a thread control memoryindexed by the corresponding thread identifier of the selected thread inresponse to the execution of a return instruction. For any of thevarious representative embodiments, the method may further comprise:generating a return work descriptor packet in response to the executionof a return instruction.

For any of the various representative embodiments, the method mayfurther comprise: generating a point-to-point event data message. Forany of the various representative embodiments, the method may furthercomprise: generating a broadcast event data message.

For any of the various representative embodiments, the method mayfurther comprise: using an event mask to respond to a received eventdata packet. For any of the various representative embodiments, themethod may further comprise: determining an event number correspondingto a received event data packet. For any of the various representativeembodiments, the method may further comprise: changing the status of athread identifier from pause to valid in response to a received eventdata packet to resume execution of a corresponding execution thread. Forany of the various representative embodiments, the method may furthercomprise: changing the status of a thread identifier from pause to validin response to an event number of a received event data packet to resumeexecution of a corresponding execution thread.

For any of the various representative embodiments, the method mayfurther comprise: successively selecting a next thread identifier fromthe execution queue for execution of a single instruction of acorresponding execution thread. For any of the various representativeembodiments, the method may further comprise: performing a round-robinselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread. For any of the variousrepresentative embodiments, the method may further comprise: performinga round-robin selection of a next thread identifier from the executionqueue, of the plurality of thread identifiers, each for execution of asingle instruction of a corresponding execution thread until completionof the execution thread. For any of the various representativeembodiments, the method may further comprise: performing a barrelselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread.

For any of the various representative embodiments, the method mayfurther comprise: assigning a valid status or a pause status to a threadidentifier. For any of the various representative embodiments, themethod may further comprise: assigning a priority status to a threadidentifier.

For any of the various representative embodiments, the method mayfurther comprise: following execution of a corresponding instruction,returning the corresponding thread identifier to the execution queuewith an assigned valid status and an assigned priority.

For any of the various representative embodiments, the method mayfurther comprise: selecting a thread identifier from a first priorityqueue at a first frequency and selecting a thread identifier from asecond priority queue at a second frequency, the second frequency lowerthan the first frequency. For any of the various representativeembodiments, the method may further comprise: determining the secondfrequency as a skip count from selection of a thread identifier from thefirst priority queue.

For any of the various representative embodiments, the method mayfurther comprise: controlling data path access size. For any of thevarious representative embodiments, the method may further comprise:increasing or decreasing memory load access size in response to timeaveraged usage levels. For any of the various representativeembodiments, the method may further comprise: increasing or decreasingmemory store access size in response to time averaged usage levels. Forany of the various representative embodiments, the method may furthercomprise: increasing a size of a memory load access request tocorrespond to a cache line boundary of the data cache.

For any of the various representative embodiments, the method mayfurther comprise: generating one or more system calls to a hostprocessor. For any of the various representative embodiments, the methodmay further comprise: using a predetermined credit count, modulating anumber of system calls in any predetermined period of time.

For any of the various representative embodiments, the method mayfurther comprise: in response to a request from a host processor,copying and transmitting all data from a thread control memorycorresponding to a selected thread identifier for monitoring threadstate.

For any of the various representative embodiments, the method mayfurther comprise: executing a fiber create instruction to generate oneor more commands to generate one or more call work descriptor packets toanother processor core or to a hybrid threading fabric circuit. For anyof the various representative embodiments, the method may furthercomprise: in response to execution of a fiber create instruction,reserving a predetermined amount of memory space for storing any returnarguments. For any of the various representative embodiments, the methodmay further comprise: in response to the generation of one or more callwork descriptor packets, storing a thread return count in the threadreturn register. For any of the various representative embodiments, themethod may further comprise: in response to receipt of a return datapacket, decrementing the thread return count stored in the thread returnregister. For any of the various representative embodiments, the methodmay further comprise: in response to the thread return count in thethread return register being decremented to zero, changing a pausedstatus to a valid status for a corresponding thread identifier forsubsequent execution of a thread return instruction for completion ofthe created fibers or threads.

For any of the various representative embodiments, the method mayfurther comprise: executing a waiting or nonwaiting fiber joininstruction. For any of the various representative embodiments, themethod may further comprise: executing a fiber join all instruction.

For any of the various representative embodiments, the method mayfurther comprise: executing a non-cached read or load instruction todesignate a general purpose register for storage of data received from amemory.

For any of the various representative embodiments, the method mayfurther comprise: executing a non-cached write or store instruction todesignate data in a general purpose register for storage in a memory.

For any of the various representative embodiments, the method mayfurther comprise: assigning a transaction identifier to any load orstore request to memory and to correlate the transaction identifier witha thread identifier.

For any of the various representative embodiments, the method mayfurther comprise: executing a first thread priority instruction toassign a first priority to an execution thread having a correspondingthread identifier. For any of the various representative embodiments,the method may further comprise: executing a second thread priorityinstruction to assign a second priority to an execution thread having acorresponding thread identifier.

For any of the various representative embodiments, the method mayfurther comprise: executing a custom atomic return instruction tocomplete an executing thread of a custom atomic operation.

For any of the various representative embodiments, the method mayfurther comprise: executing a floating point atomic memory operation.

For any of the various representative embodiments, the method mayfurther comprise: executing a custom atomic memory operation.

Numerous other advantages and features of the present invention willbecome readily apparent from the following detailed description of theinvention and the embodiments thereof, from the claims and from theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will bemore readily appreciated upon reference to the following disclosure whenconsidered in conjunction with the accompanying drawings, wherein likereference numerals are used to identify identical components in thevarious views, and wherein reference numerals with alphabetic charactersare utilized to identify additional types, instantiations or variationsof a selected component embodiment in the various views, in which:

FIG. 1 is a block diagram of a representative first embodiment of ahybrid computing system.

FIG. 2 is a block diagram of a representative second embodiment of ahybrid computing system.

FIG. 3 is a block diagram of a representative third embodiment of ahybrid computing system.

FIG. 4 is a high-level block diagram of a portion of a representativeembodiment of a hybrid threading fabric circuit cluster.

FIG. 5 is a high-level block diagram of a representative embodiment of ahybrid threading processor 300.

FIG. 6 is a detailed block diagram of a representative embodiment of athread memory of the hybrid threading processor.

FIG. 7 is a detailed block diagram of a representative embodiment of anetwork response memory of the hybrid threading processor.

FIG. 8 is a detailed block diagram of a representative embodiment of ahybrid threading processor.

FIGS. 9A and 9B (collectively FIG. 9) are a flow chart of arepresentative embodiment of a method for self-scheduling and threadcontrol for a hybrid threading processor.

FIG. 10 is a detailed block diagram of a representative embodiment of athread selection control circuitry of the control logic and threadselection circuitry of the hybrid threading processor.

FIG. 11 is a block diagram of a representative embodiment of a portionof the first interconnection network and representative data packets.

FIG. 12 is a detailed block diagram of a representative embodiment ofdata path control circuitry of a hybrid threading processor.

FIG. 13 is a detailed block diagram of a representative embodiment ofsystem call circuitry of a hybrid threading processor and host interfacecircuitry.

DETAILED DESCRIPTION OF REPRESENTATIVE EMBODIMENTS

While the present invention is susceptible of embodiment in manydifferent forms, there are shown in the drawings and will be describedherein in detail specific exemplary embodiments thereof, with theunderstanding that the present disclosure is to be considered as anexemplification of the principles of the invention and is not intendedto limit the invention to the specific embodiments illustrated. In thisrespect, before explaining at least one embodiment consistent with thepresent invention in detail, it is to be understood that the inventionis not limited in its application to the details of construction and tothe arrangements of components set forth above and below, illustrated inthe drawings, or as described in the examples. Methods and apparatusesconsistent with the present invention are capable of other embodimentsand of being practiced and carried out in various ways. Also, it is tobe understood that the phraseology and terminology employed herein, aswell as the abstract included below, are for the purposes of descriptionand should not be regarded as limiting.

FIGS. 1, 2 and 3 are block diagrams of representative first, second, andthird embodiments of a hybrid computing system 100A, 100B, 100C(collectively referred to as a system 100). FIG. 4 is a high-level blockdiagram of a portion of a representative embodiment of a hybridthreading fabric circuit cluster 205 with a second interconnectionnetwork 250. FIG. 5 is a high-level block diagram of a representativeembodiment of a hybrid threading processor (“HTP”) 300. FIG. 6 is adetailed block diagram of a representative embodiment of a thread memory320 (also referred to equivalently as a thread control memory 320 orthread context memory 320) of the HTP 300. FIG. 7 is a detailed blockdiagram of a representative embodiment of a network response memory 325of the HTP 300. FIG. 8 is a detailed block diagram of a representativeembodiment of an HTP 300. FIG. 9 is a flow chart of a representativeembodiment of a method for self-scheduling and thread control for an HTP300.

Referring to FIGS. 1-9, a hybrid computing system 100 includes a hybridthreading processor (“HTP”) 300, which is coupled through a firstinterconnection network 150 to one or more hybrid threading fabric(“HTF”) circuits 200. It should be understood that term “fabric”, asused herein, means and includes an array of computing circuits, which inthis case are reconfigurable computing circuits. FIGS. 1, 2, and 3 showdifferent system 100A, 100B, and 100C arrangements which includeadditional components forming comparatively larger and smaller systems100, any and all of which are within the scope of the disclosure. Asshown in FIGS. 1 and 2, which may each be an arrangement suitable for asystem-on-a-chip (“SOC”), for example and without limitation, a hybridcomputing system 100A, 100B, in various combinations as illustrated, mayalso include, optionally, a memory controller 120 which may be coupledto a memory 125 (which also may be a separate integrated circuit), anyof various communication interfaces 130 (such as a PCIe communicationinterface), one or more host processor(s) 110, and a host interface(“HIF”) 115. As shown in FIG. 3, which may each be an arrangementsuitable for a “chiplet” configuration on a common substrate 101, forexample and without limitation, a hybrid computing system 100C may alsoinclude, optionally, a communication interface 130, with or withoutthese other components. Any and all of these arrangements are within thescope of the disclosure, and collectively are referred to herein as asystem 100. Any of these hybrid computing systems 100 also may beconsidered a “node”, operating under a single operating system (“OS”),and may be coupled to other such local and remote nodes as well.

Hybrid threading, as used herein, refers to the capability to spawnmultiple fibers and threads of computation across different,heterogeneous types of processing circuits (hardware), such as acrossHTF circuits 200 (as a reconfigurable computing fabric) and across aprocessor, such as the HTP 300 or another type of RISC-V processor.Hybrid threading also refers to a programming language/style in which athread of work transitions from one compute element to the next to movethe compute to where the data is located, which is also implemented inrepresentative embodiments. A host processor 110 is typically amulti-core processor, which may be embedded within the hybrid computingsystem 100, or which may be an external host processor coupled into thehybrid computing system 100 via a communication interface 130, such as aPCIe-based interface. These processors, such as the HTP 300 and the oneor more host processor(s) 110, are described in greater detail below.

The memory controller 120 may be implemented as known or becomes knownin the electronic arts. Alternatively, in a representative embodiment,the memory controller 120 may be implemented as described in the relatedapplications. The first memory 125 also may be implemented as known orbecomes known in the electronic arts, and as described in greater detailbelow.

Also in a representative embodiment, the HTP 300 is a RISC-V ISA basedmulti-threaded processor having one or more processor cores 305 havingan extended instruction set, with one or more core control circuits 310and one or more second memories 315, referred to as a core controlmemories 315, as discussed in greater detail below. Generally, the HTP300 provides barrel-style, round-robin instantaneous thread switching tomaintain a high instruction-per-clock rate.

The HIF 115, for the purposes herein, provides for a host processor 110to send work to the HTP 300 and the HTF circuits 200, and for the HTP300 to send work to the HTF circuits 200, both as “work descriptorpackets” transmitted over the first interconnection network 150. Aunified mechanism is provided to start and end work on an HTP 300 and anHTF circuit 200: call work descriptor packets are utilized to start workon an HTP 300 and an HTF circuit 200, and return work descriptor packetsare utilized to end work on an HTP 300 and an HTF circuit 200. The HIF115 includes a dispatch circuit and queue (abbreviated “dispatch queue”105), which also provides management functionality for monitoring theload provided to and resource availability of the HTF circuits 200and/or HTP 300. When resources are available on the HTF circuits 200and/or HTP 300, the dispatch queue 105 determines the HTF circuit 200and/or HTP 300 resource that is least loaded. In the case of multipleHTF circuit clusters 205 with the same or similar work loading, itchooses an HTF circuit cluster 205 that is currently executing the samekernel if possible (to avoid having to load or reload a kernelconfiguration). Similar functionality of the HIF 115 may also beincluded in an HTP 300, for example, particularly for system 100arrangements which may not include a separate HIF 115. Other HIF 115functions are described in greater detail below. An HIF 115 may beimplemented as known or becomes known in the electronic arts, e.g., asone or more state machines with registers (forming FIFOs, queues, etc.).

The first interconnection network 150 is a packet-based communicationnetwork providing data packet routing between and among the HTF circuits200, the hybrid threading processor 300, and the other optionalcomponents such as the memory controller 120, a communication interface130, and a host processor 110. The first interconnection network 150 istypically embodied as a plurality of crossbar switches having a foldedclos configuration, and typically a mesh network for additionalconnections, depending upon the system 100 embodiment. For purposes ofthe present disclosure, the first interconnection network 150 forms partof an asynchronous switching fabric (“AF”), meaning that a data packetmay be routed along any of various paths, such that the arrival of anyselected data packet at an addressed destination may occur at any of aplurality of different times, depending upon the routing. This is incontrast with the synchronous mesh communication network of the secondinterconnection network 250 discussed in greater detail below. Aspectsof the first interconnection network 150 are discussed in greater detailbelow with reference to FIGS. 10 and 11.

A HTF circuit 200, in turn, typically comprises a plurality of HTFcircuit clusters 205, with each HTF circuit cluster 205 coupled to thefirst interconnection network 150 for data packet communication. EachHTF circuit cluster 205 may operate independently from each of the otherHTF circuit clusters 205. Each HTF circuit cluster 205, in turn,comprises an array of a plurality of HTF reconfigurable computingcircuits 210, which are referred to equivalently herein as “tiles” 210,and a second interconnection network 250. The tiles 210 are embedded inor otherwise coupled to the second interconnection network 250, whichcomprises two different types of networks, discussed in relatedapplications. As an overview, the HTF circuit 200 is a coarse-grainedreconfigurable compute fabric comprised of interconnected compute tiles210, for execution of a plurality of different compute operations.

The HTP 300 is a barrel style multi-threaded processor that is designedto perform well on applications with high degree of parallelismoperating on sparse data sets (i.e., applications having minimal datareuse). The HTP 300 is based on the open source RISC-V processor, andexecutes in user mode. The HTP 300 includes more RISC-V user modeinstructions, plus a set of custom instructions to allow threadmanagement, sending and receiving events to/from other HTPs 300, HTFcircuits 200 and one or more host processors 110, and instructions forefficient access to memory 125.

A processor core 305 has an associated cache memory, such as the datacache 346 illustrated as part of the core control memory 315, or whichmay be arranged internal to and/or contained within the processor core305 (not separately illustrated). Such a cache 346 is typically utilizedfor storing data which will be reused, such that the data may be fetchedfrom the cache and a memory load operation to the memory 125 is notrequired. Many applications, however, reuse very little data, and one ormore memory load operations to the memory 125 are typically required forthe application. As that data will not be reused, in representativeembodiments, the data held in such a cache 346 associated with theprocessor core 305 will not be evicted or overwritten by the datafetched from the memory 125 during the memory load operation, asdiscussed in greater detail below, but will remain available in thecache for potential reuse.

As such, sparse data sets typically cause poor cache hit rates. The HTP300 with many threads per HTP processor core 305 allows some threads tobe waiting for response from memory 125 while other threads arecontinuing to execute instructions. This style of compute is tolerant oflatency to memory 125 and allows high sustained executed instructionsper clock. The event mechanism allows threads from many HTP cores 305 tocommunicate in an efficient manner. Threads pause executing aninstruction while waiting for memory 125 responses or event messages,allowing other threads to use the instruction execution resources. TheHTP 300 is self-scheduling and event driven, allowing threads toefficiently be created, destroyed and communicate with other threads.

Work descriptor packets are utilized to commence work on an HTP 300 anda HTF circuit 200. Receipt of a work descriptor packet by an HTP 300constitutes an “event” which will trigger hardware-based self-schedulingand subsequent execution of the associated functions or work, referredto as threads of execution, in the HTP 300, without the need for furtheraccess to main memory 125. Once a thread is started it executesinstructions until a thread return instruction is executed. The threadreturn instruction sends a return work descriptor packet to the originalcaller.

For purposes of the present disclosure, at a high or general level, awork descriptor packet includes the information needed to initialize athread context for the HTP 300, such as a program count (e.g., as a64-bit address) for where in the stored instructions (stored ininstruction cache 340) to commence thread execution, and any argumentsor addresses in first memory 125 to obtain arguments or otherinformation which will be used in the thread execution, and a returnaddress for transmission of computation results, for example and withoutlimitation. There can be many different kinds of work descriptorpackets, depending upon the operations or instructions to be performed,with many examples illustrated and discussed below. The instructioncache 340 has been populated in advance of any execution, such as in theinitial system 100 configuration.

Accordingly, in many instances, the HTP 300 allows threads to be createdand execution started without a single access to main memory 125. This“light weight” thread creation allows many threads to be created when anapplication's parallel region is entered with minimal delay and very lowlatency, in sharp contrast with prior art computing architectures, asthread creation is done by initializing a small (64B) work descriptorpacket in hardware, then sending that packet to the destination HTP 300where a thread is to be started. The receiving HTP 300 takes the packetand initialize a thread's hardware context from the work descriptorpacket. The thread is immediately started executing instructions. Asmentioned above, the work descriptor packet contains only theinstruction PC where execution is to start and some number of callarguments (e.g., up to four). The receiving HTP 300 initializes theremainder of the thread context state autonomously in preparation forstarting the thread executing instructions.

An executing thread has memory stack space and main memory 125 contextspace. The context space is only used if the state of the thread needsto be written to memory 125 to be accessed by the host processor 110.Each HTP 300 is initialized with a core stack base address and a corecontext base address, where the base addresses point a block of stacksand a block of context spaces. The thread stack base address is obtainedby taking the core stack base address and adding the thread IDmultiplied by the thread stack size. The thread context base address isobtained in a similar fashion.

An HTP 300 typically comprises one or more processor cores 305 which maybe any type of processor core, such as a RISC-V processor core, an ARMprocessor core, etc., all for example and without limitation. A corecontrol circuit 310 and a core control memory 315 are provided for eachprocessor core 305, and are illustrated in FIG. 5 for one processor core305. For example, when a plurality of processor cores 305 areimplemented, such as in one or more HTPs 300, corresponding pluralitiesof core control circuits 310 and core control memories 315 are alsoimplemented, with each core control circuit 310 and core control memory315 utilized in the control of a corresponding processor core 305. Inaddition, one or more of the HTPs 300 may also include data path controlcircuitry 395, which is utilized to control access sizes (e.g., memory125 load requests) over the first interconnection network 150 to managepotential congestion of the data path.

In turn, a core control circuit 310 comprises control logic and threadselection circuitry 330 and network interface circuitry 335. The corecontrol memory 315 comprises a plurality of registers or other memorycircuits, conceptually divided and referred to herein as thread memory(or thread control memory) 320 and network response memory 325. Thethread memory 320 includes a plurality of registers to store informationpertaining to thread state and execution, while the network responsememory 325 includes a plurality of registers to store informationpertaining to data packets transmitted to and from first memory 125 onthe first interconnection network 150, such as requests to the firstmemory 125 for reading or storing data, for example and withoutlimitation.

Referring to FIG. 6, the thread memory 320 includes a plurality ofregisters, including thread ID pool registers 322 (storing apredetermined number of thread IDs which can be utilized, and typicallypopulated when the system 100 is configured, such as with identificationnumbers 0 to 31, for a total of 32 thread IDs, for example and withoutlimitation); thread state (table) registers 324 (storing threadinformation such as valid, idle, paused, waiting for instruction(s),first (normal) priority, second (low) priority, temporary changes topriority if resources are unavailable); program counter registers 326(e.g., storing an address or a virtual address for where the thread iscommencing next in the instruction cache 340); general purpose registers328 for storing integer and floating point data; pending fiber returncount registers 332 (tracking the number of outstanding threads to bereturned to complete execution); return argument buffers 334 (“RAB”,such as a head RAB as the head of a link list with return argumentbuffers), thread return registers 336 (e.g., storing the return address,a call identifier, any thread identifier associated with the callingthread); custom atomic transaction identifier(s) registers 338; eventreceived mask registers 342 (to designate which events to “listen” for,as discussed in greater detail below), event state registers 344, and adata cache 346 (typically providing 4-8 cache lines of cache memory foreach thread). All of the various registers of the thread memory 320 areindexed using the assigned thread ID for a given or selected thread.

Referring to FIG. 7, the network response memory 325 includes aplurality of registers, such as memory request (or command) registers348 (such as commands to read, write, or perform a custom atomicoperation); thread ID and transaction identifiers (“transaction IDs”)registers 352 (with transaction IDs utilized to track any requests tomemory, and associating each such transaction ID with the thread ID forthe thread which generated the request to memory 125); a request cacheline index register 354 (to designate which cache line in the data cache346 is to be written to when data is received from memory for a giventhread (thread ID)), register bytes register 356 (designating the numberof bytes to write to the general purpose registers 328); and a generalpurpose register index and type registers 358 (indicating which generalpurpose register 328 is to be written to, and whether it is signextended or floating point).

As described in greater detail below, an HTP 300 will receive a workdescriptor packet. In response, the HTP 300 will find an idle or emptycontext and initialize a context block, assigning a thread ID to thatthread of execution (referred to herein generally as a “thread”), if athread ID is available, and puts that thread ID in a an execution (i.e.,“ready-to-run”) queue 345. Threads in the execution (ready-to-run) queue345 are selected for execution, typically in a round-robin or “barrel”style selection process, with a single instruction for the first threadprovided to the execution pipeline 350 of the processor core 305,followed by a single instruction for the second thread provided to theexecution pipeline 350, followed by a single instruction for the thirdthread provided to the execution pipeline 350, followed by a singleinstruction for the next thread provided to the execution pipeline 350,and so on, until all threads in the execution (ready-to-run) queue 345have had a corresponding instruction provided to the execution pipeline350, at which point the thread selection commences again with a nextinstruction for the first thread in the execution (ready-to-run) queue345 provided to the execution pipeline 350, followed by a nextinstruction for the second thread provided to the execution pipeline350, and so on, cycling through all of the threads of the execution(ready-to-run) queue 345. This execution will continue for each suchthread until execution for that thread has been completed, such as byexecuting a thread return instruction, at which point a response packet(having the results of the thread execution) is transmitted back to thesource of the work descriptor packet, i.e., back to the source of thework descriptor call packet. In addition, in a representative embodimentand as discussed in greater detail below, the execution (ready-to-run)queue 345 is optionally provided with different levels of priority,illustrated as a first priority queue 355 and a second (lower) priorityqueue 360, with execution of the threads in the first priority queue 355occurring more frequently than the execution of the threads in thesecond (lower) priority queue 360.

As a result, the HTP 300 is an “event driven” processor, and willautomatically commence thread execution upon receipt of a workdescriptor packet (provided a thread ID is available, but without anyother requirements for initiating execution), i.e., arrival of a workdescriptor packet automatically triggers the start of thread executionlocally, without any reference to or additional requests to memory 125.This is tremendously valuable, as the response time to commenceexecution of many threads in parallel, such as thousands of threads, iscomparatively low. The HTP 300 will continue thread execution untilthread execution is complete, or it is waiting for a response, at whichpoint that thread will enter a “pause” state, as discussed in greaterdetail below. A number of different pause states are discussed ingreater detail below. Following receipt of that response, the thread isreturned to an active state, at which point the thread resumes executionwith its thread ID returned to the execution (ready-to-run) queue 345.This control of thread execution is performed in hardware, by thecontrol logic and thread selection circuitry 330, in conjunction withthread state information stored in the thread memory 320.

In addition to a host processor 110 generating work descriptor packets,an HTP 300 can also generate and transmit work descriptor packets toinitiate work, as one or more compute threads, on another computingresource, such as another HTP 300 or any HTF circuit 200. Such a workdescriptor packet is a “call” work descriptor packet, and generallycomprises a source identifier or address for the host processor 110 orthe HTP 300 which is generating the call work descriptor packet, athread ID (such as a 16-bit call identifier (ID)) used to identify orcorrelate the return with the original call, a 64-bit virtual kerneladdress (as a program count, to locate the first instruction to beginexecution of the thread, typically held in the instruction cache 340 ofan HTP 300 (or of a HTF circuit 200), which also may be a virtualaddress space), and one or more call arguments, e.g., up to four callarguments).

Similarly, when the thread has been completed, the HTP 300 or HTFcircuit 200 generates another work descriptor packet, referred to as a“return” work descriptor packet, which is generally created when the HTP300 or HTF circuit 200 executes the last instruction of the thread,referred to as a return instruction, with the return work descriptorpacket assembled by the packet encoder 380, discussed below. The returnpacket will be addressed back to the source (using the identifier oraddress provided in the call work descriptor packet), the thread ID (orcall ID) from the call work descriptor packet (to allow the source tocorrelate the return with the issued call, especially when multiplecalls have been generated by the source and are simultaneouslyoutstanding), and one or more return values (as results), such as up tofour return values.

FIG. 8 is a detailed block diagram of a representative embodiment of anHTP 300. For ease of illustration and discussion, it should be notedthat not all registers of the thread memory 320 and the network responsememory 325 are illustrated in FIG. 8. Referring to FIG. 8, the corecontrol circuit 310 comprises control logic and thread selectioncircuitry 330 and network interface circuitry 335. The control logic andthread selection circuitry 330 comprises circuitry formed usingcombinations of any of a plurality of various logic gates (e.g., NAND,NOR, AND, OR, EXCLUSIVE OR, etc.) and various state machine circuits(control logic 331), and multiplexers (e.g., input multiplexer 387,thread selection multiplexer 385), for example and without limitation.The network interface circuitry 335 includes AF input queues 365 toreceive data packets (including work descriptor packets) from the firstinterconnection network 150; AF output queues 370 to transfer datapackets (including work descriptor packets) to the first interconnectionnetwork 150; a data packet decoder circuit 375 to decode incoming datapackets from the first interconnection network 150, take data (indesignated fields) and transfer the data provided in the packet to therelevant registers of the thread memory 320 and the network responsememory 325 (in conjunction with the thread ID assigned to the thread bythe control logic and thread selection circuitry 330, as discussed ingreater detail below, which thread ID also provides or forms the indexto the thread memory 320); and data packet encoder circuit 380 to encodeoutgoing data packets (such as requests to memory 125, using atransaction ID from thread ID and transaction identifiers (“transactionIDs”) registers 352) for transmission on the first interconnectionnetwork 150. The data packet decoder circuit 375 and the data packetencoder circuit 380 may each be implemented as state machines or otherlogic circuitry.

When a work descriptor packet arrives, the control logic and threadselection circuitry 330 assigns an available thread ID to the thread ofthe word descriptor packet, from the thread ID pool registers 322, withthe assigned thread ID used as an index to the other registers of thethread memory 320 which are then populated with corresponding data fromthe work descriptor packet, typically the program count and one or morearguments. The control logic and thread selection circuitry 330initializes the remainder of the thread context state autonomously inpreparation for starting the thread executing instructions, such asloading the data cache registers 346 and loading the thread returnregisters 336, for example and without limitation. Also for example, anexecuting thread has main memory stack space and main memory contextspace. The context space is only used if the state of the thread needsto be written to memory to be accessed by the host. Each HTP 300processor core 305 is initialized with a core stack base address and acore context base address, where the base addresses point a block ofstacks and a block of context spaces. The thread stack base address isobtained by taking the core stack base address and adding the thread IDmultiplied by the thread stack size. The thread context base address isobtained in a similar fashion.

That thread ID is given a valid status (indicating it is ready toexecute), and the thread ID is pushed to the first priority queue 355 ofthe execution (ready-to-run) queue(s) 345, as threads are typicallyassigned a first (or normal) priority. Selection circuitry of thecontrol logic and thread selection circuitry 330, such as a multiplexer385, selects the next thread ID in the execution (ready-to-run) queue(s)345, which is used as an index into the thread memory 320 (the programcount registers 326 and thread state registers 324), to select theinstruction from the instruction cache 340 which is then provided to theexecution pipeline 350 for execution. The execution pipeline thenexecutes that instruction.

At completion of execution of the instruction, under the control of thecontrol logic and thread selection circuitry 330 the same triplet ofinformation (thread ID, valid state, and priority) can be returned tothe execution (ready-to-run) queue(s) 345, for continued selection forround-robin execution, depending upon various conditions. For example,if the last instruction for a selected thread ID was a returninstruction (indicating that thread execution was completed and a returndata packet is being provided), the control logic and thread selectioncircuitry 330 will return the thread ID to the available pool of threadIDs in the thread ID pool registers 322, to be available for use byanother, different thread. Also for example, the valid indicator couldchange, such as changing to a pause state (such as while the thread maybe waiting for information to be returned from or written to memory 125or waiting for another event), and in which case, the thread ID (nowhaving a pause status) is not returned to the execution (ready-to-run)queue(s) 345 until the status changes back to valid.

Continuing with the former example, when the last instruction for aselected thread ID was a return instruction, the return information(thread ID and return arguments) is then pushed by the executionpipeline 350 to the network command queue 390, which is typicallyimplemented as first-in, first out (FIFO). The thread ID is used as anindex into the thread return registers 336 to obtain the returninformation, such as the transaction ID and the source (caller) address(or other identifier), and the packet encoder circuit 380 then generatesan outgoing return data packet (on the first interconnection network150).

Continuing with the latter example, an instruction of a thread may be aload instruction, i.e., a read request to the memory 125, which is thenpushed by the execution pipeline 350 to the network command queue 390.The packet encoder circuit 380 then generates an outgoing data packet(on the first interconnection network 150) with the request to memory125 (as either a read or a write request), including the size of therequest and an assigned transaction ID (from the thread ID andtransaction IDs registers 352, which is also used as an index into thenetwork response memory 325), the address of the HTP 300 (as the returnaddress of the requested information). When a packet is then receivedfrom the first interconnection network 150 and decoded, the transactionID is used as an index into the network response memory 325, the threadID of the thread which made the request is obtained, which also providesthe location in the data cache 346 to write the data returned in theresponse, with the transaction ID then returned to the thread ID andtransaction ID registers 352 to be reused, and the status of thecorresponding thread ID is set again to valid and the thread ID is againpushed to the execution (ready-to-run) queue(s) 345, to resumeexecution.

A store request to memory 125 is executed similarly, with the outgoingpacket also having the data to be written to memory 125, an assignedtransaction ID, the source address of the HTP 300, and with the returnpacket being an acknowledgement with the transaction ID. The transactionID is also then returned to the thread ID and transaction ID registers352 to be reused, and the status of the corresponding thread ID is setagain to valid and the thread ID is again pushed to the execution(ready-to-run) queue(s) 345, to resume execution.

FIG. 9 is a flow chart of a representative embodiment of a method forself-scheduling and thread control for an HTP 300, and provides a usefulsummary, with the HTP 300 having already been populated withinstructions in the instruction cache 340 and a predetermined number ofthread IDs in the thread identifier pool register 322. The methodstarts, step 400, upon reception of a work descriptor packet. The workdescriptor packet is decoded, step 402, and the various registers of thethread memory 320 are populated with the information received in thework descriptor packet, initializing a context block, step 404. When athread ID is available, step 406, a thread ID is assigned, step 408 (andif a thread ID is not available in step 406, the thread will wait untila thread ID becomes available, step 410). A valid status is initiallyassigned to the thread (along with any initially assigned priority, suchas a first or second priority), step 412, and the thread ID is providedto the execution (ready-to-run) queue 345, step 414. A thread ID in theexecution (ready-to-run) queue 345 is then selected for execution (at apredetermined frequency, discussed in greater detail below), step 416.Using the thread ID, the thread memory 320 is accessed, and a programcount (or address) is obtained, step 418. The instruction correspondingto the program count (or address) is obtained from the instruction cache340 and provided to the execution pipeline 350 for execution, step 420.

When the thread execution is complete, i.e., the instruction beingexecuted is a return instruction, step 422, the thread ID is returned tothe thread ID pool registers 322 for reuse by another thread, step 424,the thread memory 320 registers associated with that thread ID may becleared (optionally), step 426, and the thread control may end for thatthread, return step 434. When the thread execution is not complete instep 422, and when the thread state remains valid, step 428, the threadID (with its valid state and priority) is returned to the execution(ready-to-run) queue 345, returning to step 414 for continued execution.When the thread state is no longer valid (i.e., the thread is paused) instep 428, with the paused status for that thread ID indicated in thethread memory 320, execution of that thread is suspended, step 430,until the status for that thread ID returns to valid, step 432, and thethread ID (with its valid state and priority) is returned to theexecution (ready-to-run) queue 345, returning to step 414 for continuedexecution.

Similarly, the HTP 300 may generate calls, such as to create threads onlocal or remote compute elements, such as to create threads on otherHTPs 300 or HTF circuits 200. Such calls are also created as outgoingdata packets, and more specifically as outgoing work descriptor packetson the first interconnection network 150. For example, an instruction ofa current thread being executed may be a “fiber create” instruction(stored as a possible instruction in the instruction cache 340), tospawn a plurality of threads for execution on the various computeresources. As discussed in greater detail below, such a fiber createinstruction designates (using an address or virtual address (nodeidentifier)) what computing resource(s) will execute the threads, andwill also provide associated arguments. When the fiber createinstruction is executed in the execution pipeline 350, the fiber createcommand is pushed into the network command queue 390, and the nextinstruction is executed in the execution pipeline 350. The command ispulled out of the network command queue 390, and the packet encoder 380has the information needed to create and send a work descriptor packetto the specified destination HTF 200 or HTP 300.

If the created threads will have return arguments, then such aninstruction will also allocate and reserve associated memory space, suchas in the return argument buffers 334. If there is insufficient space inthe return argument buffers 334, the instruction will be paused untilreturn argument buffers 334 are available. The number of fibers orthreads created is only limited by the amount of space to hold theresponse arguments. Created threads that do not have return argumentscan avoid reserving return argument space, avoiding the possible pausestate. This mechanism ensures that returns from completed threads alwayshave a place to store their arguments. As the returns come back to theHTP 300 as data packets on the first interconnection network 150, thosepackets are decoded, as discussed above, with the return data stored inthe associated, reserved space in the return argument buffers 334 of thethread memory 320, as indexed by the thread ID associated with the fibercreate instruction. As many registers could be utilized for the returnargument, the return argument buffers 334 can be provided as a link listof all the spawned threads or return argument buffers or registersallocated for that thread ID. Significantly, this mechanism can allowpotentially thousands of threads to be created very quickly, effectivelyminimizing the time involved in a transition from a single threadexecution to high thread count parallelism.

As discussed in greater detail below, various types of fiber joininstructions are utilized to determine when all of the spawned threadshave completed, and can be an instruction with or without waiting. Acount of the number of spawned threads is maintained in the pendingfiber return count registers 332, which count is decremented as threadreturns are received by the HTP 300. A join operation can be carried outby copying the returns into the registers associated with the spawningthread ID. If the join instruction is a waiting instruction, it willstay in a paused state until the return arrives which designates thatthread ID of the spawning thread. In the interim, other instructions areexecuted by the execution pipeline 350 until the pause state of the joininstruction changes to a valid state and the join instruction isreturned to the execution (ready-to-run) queue 345.

A thread return instruction may also be utilized as the instructionfollowing the fiber create instruction, instead of a join instruction.When the count in the pending fiber return count registers 332 reacheszero, with the receipt of the last thread return data packet, a threadreturn instruction may also be executed, and indicates that the fibercreate operation has been completed and all returns received, allowingthe thread ID, the return argument buffers 334, and link list to befreed for other uses. In addition, it may also generate and transmit awork descriptor return packet (e.g., having result data) to the sourcewhich called the main thread (e.g., to the identifier or address of thesource which generated the call).

The join all instruction does not require that arguments be returned,only acknowledgements which decrement the count in the pending fiberreturn count registers 332. When that count reaches zero, that thread isrestarted, as the join all is now complete.

Communication between processing elements is required to facilitateprocessing of parallel algorithms. The representative embodimentsprovide an efficient means for threads of a set of processing resourcesto communicate, using various event messages—which may also include data(such as arguments or results). The event messaging allows any hostprocessors 110 with hardware maintained cache coherency and anyacceleration processors (such as the HTP 300) with software maintainedcache coherency to efficiently participate in event messaging.

The event messaging supports both point to point and broadcast eventmessages. Each processing resource (HTP 300) can determine when areceived event operation has completed and the processing resourceshould be informed. The event receive modes include simple (a singlereceived event completes the operation), collective (a counter is usedto determine when sufficient events have been received to complete theoperation), and broadcast (an event received on a specific channelcompletes the event). Additionally, events can be sent with an optional64-bit data value.

The HTP 300 has a set of event receive states, stored in the event stateregisters 344, that consists of a 2-bit receive mode, a 16-bitcounter/channel number, and a 64-bit event data value. An HTP 300 canhave multiple sets of event receive states per thread context, whereeach set is indexed by an event number. Thus, an event can be targetedto a specific thread (thread ID) and event number. The sent event can bea point-to-point message with a single destination thread, or abroadcast message sent to all threads within a group of processingresources belonging to the same process. When such events are received,the paused or sleeping thread can be reactivated to resume processing.

This use of event state registers 344 is much more efficient than astandard Linux based host processor, which can send and receive eventsthrough an interface that allows the host processor 110 to periodicallypoll on completed receive events. Threads waiting on event messages canpause execution until the receive operation completes, i.e., the HTP 300can pause execution of threads pending the completion of receive events,rather than waste resources by polling, allowing other threads to beexecuting during these intervals. Each HTP 300 also maintains a list ofprocessing resources that should participate in receiving events toavoid process security issues.

A point-to-point message will specify an event number and thedestination (e.g., node number, which HTP 300, which core, and whichthread ID). On the receive side, an HTP 300 will have been configured orprogrammed with one or more event numbers held in the event stateregisters 344. If that HTP 300 receives an event message having thatevent number, it is triggered and transitions from a paused state to avalid state to resume execution, such as executing an event receivedinstruction (e.g., EER, below). That instruction will then determine ifthe correct event number was received, and if so, write any associated64-bit data into general purpose registers 328, for use by anotherinstruction. If the event received instruction executes and the correctevent number was not received, it will be paused until that specificevent number is received.

An event listen (EEL) instruction may also be utilized, with an eventmask stored in the event received mask registers 342, indicating one ormore events which will be used to trigger or wake up the thread. When anevent message with any of those designated events arrives, the receivingHTP 300 will know which event number was triggered, e.g., what otherprocess may have been completed, and will receive event data from thosecompleted events. The event listen instruction may also have waiting anda no waiting variations, as discussed in greater detail below.

For event messaging in a collective mode, the receiving HTP 300 willcollect (wait for) a set of receive events before triggering, setting acount in the event state registers 344 to the value required, which isdecremented as the required event messages are received, and triggeringonce the count has been decremented to zero.

In a broadcast mode, a sender processing resource can transmit a messageto any thread within the node. For example, a sending HTP 300 maytransmit a series of point-to-point messages to each other HTP 300within the node, and each receiving HTP 300 will then pass the messageto each internal core 305. Each core control circuit 310 will go throughits thread list to determine if it corresponds to an event number whichit has been initialized to receive, and upon which channel that may havebeen designated on the first interconnection network 150.

This broadcast mode is especially useful when thousands of threads maybe executing in parallel, in which the last thread to execute transmitsa broadcast event message indicating completion. For example, a firstcount of all threads requiring completion may be maintained in the eventstate registers 344, while a second count of all threads which haveexecuted may be maintained in memory 125. As each thread executes, italso performs a fetch and increment atomic operation on the secondcount, such as through an atomic operation of the memory 125 (andcompares it to the first count), and sets its mode to receive abroadcast message by executing an EER instruction to wait until itreceives a broadcast message. The last one to execute will see thefetched value of the second count as the required first count minus one,indicating that it is the last thread to execute, and therefore sendsthe broadcast message, which is a very fast and efficient way toindicate completion of significant parallel processing.

As mentioned above, while the HTP 300 may utilize standard RISC-Vinstructions, a significantly extended set of instructions are providedto take advantage of all the system 100 compute resources, as discussedin greater detail below. Threads created from the host processor 110 aretypically referred to as master threads, and threads created from theHTP 300 are typically referred to as fibers or fiber threads, and allare executed identically on the destination HTP 300 and HTF 200, withoutgoing through the memory 125.

Load Instructions:

The HTP 300 has a comparatively small number of read/write buffers perthread, also referred to as data cache registers 346. The buffers (datacache registers 346) temporarily store shared memory data for use by theowning thread. The data cache registers 346 are managed by a combinationof hardware and software. Hardware automatically allocates buffers andevicts data when needed. Software, through the use of RISC-Vinstructions decides which data should be cached (read and write data),and when the data cache registers 346 should be invalidated (if clean)or written back to memory (if dirty). The RISC-V instruction setprovides a FENCE instruction as well as acquire and release indicatorson atomic instructions.

The standard RISC-V load instructions automatically use the read datacache registers 346. A standard load checks to see if the needed data isin an existing data cache register 346. If it is then the data isobtained from the data cache register 346 and the executing thread isable to continue execution without pausing. If the needed data is not ina data cache register 346, then the HTP 300 finds an available datacache register 346 (evicting data from a buffer needed), and reads 64bytes from memory into the data cache register 346. The executing threadis paused until the memory read has completed and the load data iswritten into a RISC-V register.

Read buffering has two primary benefits: 1) larger accesses are moreefficient for the memory controller 120, and 2) accesses to the bufferallow the executing thread to avoid stalling. However, there aresituations when using the buffer causes problems. An example is a gatheroperation where accesses would typically cause thrashing of the datacache registers 346. For this reason, a set of special load instructionsare provided to force a load instruction to check for a cache hit, buton a cache miss to issue a memory request for just the requested operandand not put the obtained data in a data cache register 346, and insteadput the data into one of the general purpose registers 328.

These load instruction provides for “probabilistic” caching based uponanticipated frequency of access, for frequently used data versussparsely or rarely used data. This is especially significant for usewith sparse data sets, which if put into the data cache registers 346,would overwrite other data which will be needed again more frequently,effectively polluting the data cache registers 346. The load instruction(NB or NC) allows frequently used data to remain in the data cacheregisters 346, and less frequently used (sparse) data which would betypically cached to be designated instead for non-cached storage in thegeneral purpose registers 328.

Instructions of this type have an NB suffix (non-buffered) (orequivalently, an NC suffice (non-cached)):

-   -   LB.NB RA,40(SP).        The NB (NC) load instructions are expected to be used in runtime        libraries written in assembly.

The following load instructions were added as 32-bit instructions, whereImm is the immediate field, RA is a register name, rs1 is a sourceindex, rd is a destination index, and the bits in fields 14-12 and 6-0specify the instruction, in Table 1.

TABLE 1 31 20 19 15 14 12 11 7 6 0 Imm[11:0] rs1 000 rd 0000010 LB.NBImm[11:0] rs1 001 rd 0000010 LH.NB Imm[11:0] rs1 010 rd 0000010 LW.NBImm[11:0] rs1 011 rd 0000010 LD.NB Imm[11:0] rs1 100 rd 0000010 LBU.NBImm[11:0] rs1 101 rd 0000010 LHU.NB Imm[11:0] rs1 110 rd 0000110 LWU.NBImm[11:0] rs1 010 rd 0000110 FLW.NB Imm[11:0] rs1 011 rd 0000110 FLD.NB

Bandwidth to memory is often the major contributor to limiting anapplication's performance. The representative embodiments provides ameans to inform the HTP 300 as to how large of a memory load requestshould be issued to memory 125. The representative embodiments reducewasted memory and bandwidth of the first interconnection network 150 dueto access memory data that is not used by the application.

A further optimization exists where an application knows the size of adata structure being accessed and can specify the amount of data to beloaded into a data cache register 346. As an example, if an algorithmuses a structure that is 16 bytes in size, and the structures arescattered in memory, then it would be optimal to issue 16 byte memoryreads and place the data into a data cache register 346. Therepresentative embodiments define a set of memory load instructions thatprovide both the size of the operand to be loaded into an HTP 300register, and the size of the access to memory if the load misses thedata cache register 346. The actual load to memory 125 may be smallerthan the instruction specified size if the memory access would cross acache line boundary. In this case, the access size is reduced to ensurethat the response data is written to a single cache line of the datacache registers 346.

When the requested data would be less than a cache line, the loadinstruction may also request additional data that the HTP 300 iscurrently unneeded but likely to be needed in the future, which is worthobtaining at the same time (e.g., as a pre-fetch), optimizing the readsize access to memory 125. This instruction can also override anyreductions in access size which might have been utilized (as discussedin greater detail below with reference to FIG. 12) for bandwidthmanagement.

The representative embodiments therefore minimize wasted bandwidth byonly requesting memory data that is known to be needed. The result is anincrease in application performance.

A set of load instructions have been defined that allow the amount ofdata to be accessed to be specified. The data is written into a buffer,and invalidated by an eviction, a FENCE, or an atomic with acquirespecified. The load instructions provide hints as to how much additionaldata (in 8-byte increments) is to be accessed from memory and written tothe memory buffer. The load will only access additional data to the next64 byte boundary. A load instruction specifies the number of additional8-byte elements to load using the operation suffix RB0-RB7:

-   -   LD.RB7 RA,40(SP)

The instruction formats are shown in Table 2. The number of 8-byte dataelements to load into the buffer is specified as bits 6 and 4:3 of the32-bit instruction. These load instructions can be used in assemblywritten routines, or ideally by a compiler. It is expected thatinitially only hand written assembly will take advantage of theseinstructions.

TABLE 2 31 20 19 15 14 12 11 7 6 0 Imm[11:0] rs1 000 rd x0xx010 LB.RC1-7Imm[11:0] rs1 001 rd x0xx010 LH.RC1-7 Imm[11:0] rs1 010 rd x0xx010LW.RC1-7 Imm[11:0] rs1 011 rd x0xx010 LD.RC1-7 Imm[11:0] rs1 100 rdx0xx010 LBU.RC1-7 Imm[11:0] rs1 101 rd x0xx010 LHU.RC1-7 Imm[11:0] rs1110 rd x0xx010 LWU.RC1-7 Imm[11:0] rs1 010 rd x0xx110 FLW.RC1-7Imm[11:0] rs1 011 rd x0xx110 FLD.RC1-7

Store Instructions

The HTP 300 has a small number of memory buffers that temporarily storeshared memory data. The memory buffers allow multiple writes to memoryto be consolidated into a smaller number of memory write requests. Thishas two benefits: 1) the fewer write requests is more efficient for thefirst interconnection network 150 and memory controllers 120, and 2) anHTP 300 suspends the thread that performs a memory store until the datais stored to either the HTP 300 memory buffer, or at the memorycontroller 120. Stores to the HTP 300 memory buffer are very quick andwill typically not cause the thread to suspend execution. When a bufferis written to the memory controller 120, then the thread is suspendeduntil a completion is received in order to ensure memory 125consistency.

The standard RISC-V store instructions write data to the HTP 300 memorybuffers. However, there are situations in which it is known that it isbetter to write the data directly to memory and not write to a memorybuffer. One such situation is a scatter operation. A scatter operationwould typically write just a single data value to the memory buffer.Writing to the buffer causes the buffers to thrash and other store datathat would benefit from write coalescing is forced back to memory. A setof store instructions are defined for the HTP 300 to indicate that writebuffering should not be used. These instructions write data directly tomemory 125, causing the executing thread to be paused until the writecompletes.

The store no buffering instructions are expected to be used in handassembled libraries and are indicated with a NB suffix:

-   -   ST.NB RA,40(SP)        The following store instructions were added as shown in Table 3.

TABLE 3 31 25 24 20 19 15 14 12 11 7 6 0 Imm[11:5] rs2 rs1 000 Imm[4:0]0100010 SB.NB Imm[11:5] rs2 rs1 001 Imm[4:0] 0100010 SH.NB Imm[11:5] rs2rs1 010 Imm[4:0] 0100010 SW.NB Imm[11:5] rs2 rs1 011 Imm[4:0] 0100010SD.NB Imm[11:5] rs2 rs1 010 Imm[4:0] 0100110 FSW.NB Imm[11:5] rs2 rs1011 Imm[4:0] 0100110 FSD.NB

Custom Atomic Store and Clear Lock (CL) Instructions:

Custom atomic operations set a lock on the provided address when theatomic operation is observed by the memory controller. The atomicoperation is performed on an associated HTP 300. The HTP 300 shouldinform the memory controller when the lock should be cleared. Thisshould be on the last store operation that the HTP 300 performs for thecustom atomic operation (or on a fiber terminate instruction if no storeis required). The HTP 300 indicates that the lock is to be cleared byexecuting a special store operation.

The following sequence of instructions could be used to implement acustom atomic DCAS operation.

 // a0 - atomic address  // a1 - 64-bit memory value of a0  // a2 - DCAScompare value 1  // a3 - DCAS compare value 2  // a4 - DCAS swap value 1 // a5 - DCAS swap value 2 atomic_dcas:  bne a1, a2, fail  // first8-byte compare  ld.nb a6, 8(a0)  //load second 8-byte  memoryvalue-should hit memory cache  bne a6, a3, fail  // second 8-bytecompare  sd a4, 0(a0)  // store first 8-byte swap value  to thread storebuffer  sd.cl a5, 8(a0)  // store second 8-byte value  and clear memorylock  eft x0  // AMO success response fail:  li a1, 1  eft.cl a1, (a0) // AMO failure response  (and clear memory lock) atomic_float_add: fadd.d a2, a1, a2  // a1 contains memory value,  a2 contains value tobe added in  fsd.cl a2, 0(a0)  // a0 contains memory address,  clearlock and terminate atomic eft // evict all line from buffer, terminateatomic threadThe store instructions that indicate the lock should be cleared are:

-   -   SB.CL RA,40(SP)    -   SH.CL RA,40(SP)    -   SW.CL RA,40(SP)    -   SD.CL RA,40(SP)    -   FSW.CL RA,40(SP)    -   FSD.CL RA,40(SP)        The format for these store instructions is shown Table 4.

TABLE 4 31 25 24 20 19 15 14 12 11 7 6 0 Imm[11:5] rs2 rs1 000 Imm[4:0]0110010 SB.CL Imm[11:5] rs2 rs1 001 Imm[4:0] 0110010 SH.CL Imm[11:5] rs2rs1 010 Imm[4:0] 0110010 SW.CL Imm[11:5] rs2 rs1 011 Imm[4:0] 0110010SD.CL Imm[11:5] rs2 rs1 010 Imm[4:0] 0110110 FSW.CL Imm[11:5] rs2 rs1011 Imm[4:0] 0110110 FSD.CL

Fiber Create Instructions:

The Fiber Create (“EFC”) instruction initiates a thread on an HTP 300 orHTF 200.

-   -   EFC.HTP.A4    -   EFC.HTF.A4        This instruction performs a call on an HTP 300 (or HTF 200),        begins execution at the address in register a0. (Optionally, a        suffix .DA may be utilized. The instruction suffix DA indicates        that the target HTP 300 is determined by the virtual address in        register a1. If the DA suffix is not present, then an HTP 300 on        the local system 100 is targeted.) The suffix A1, A2 and A4        specifies the number of additional arguments to be passed to the        HTP 300 or HTF 200. The argument count is limited to the values        0, 1, 2, or 4 (e.g., a packet should fit in 64B). The additional        arguments are from register state (a2-a5).

It should be noted that if a return buffer is not available at the timethe EFC instruction is executed, then the EFC instruction will waituntil a return argument buffer is available to begin execution. Once theEFC instruction successfully creates a fiber, the thread continues atthe instruction immediately following the EFC instruction.

It also should be noted that threads created by the host processor 110are allowed to execute the EFC instruction and create fibers. Fiberscreated by an EFC instruction are not allowed to execute the EFCinstruction and will force an exception, optionally. The format forthese fiber create instructions is shown Table 5.

TABLE 5 31 25 24 20 19 15 14 12 11 7 6 0 0000000 00000 00000 ac 000001110010 EFC.HTP 0001000 00000 00000 ac 00000 1110010 EFC.HTP.DA 001000000000 00000 ac 00000 1110010 EFC.HTF 0011000 00000 00000 ac 000001110010 EFC.HTF.DA

ac Encoding Suffix Argument Count 0 No suffix 0 1 .A1 1 2 .A2 2 3 .A4 4

Thread Return Instructions:

The Thread Return (ETR) instruction passes arguments back to the parentthread that initiated the current thread (through a host processor 110thread create or HTP 300 fiber create). Once the thread has completedthe return instruction, the thread is terminated.

-   -   ETR.A2        This instruction performs a return to an HTP 300 or host        processor 110. The ac suffix specifies the number of additional        arguments to be passed to the HTP or host. Argument count can be        the values 0, 1, 2 or 4. The arguments are from register state        (a0-a3). The format for these thread return instructions is        shown Table 6.

TABLE 6 31 25 24 20 19 15 14 12 11 7 6 0 0100000 00000 00000 ac 000001110010 EFR

ac Encoding Suffix Argument Count 0 No suffix 0 1 .A1 1 2 .A2 2 3 .A4 4

Fiber Join Instructions:

The Fiber Join (EFJ) instruction checks to see if a created fiber hasreturned. The instruction has two variants, join wait and non-wait. Thewait variant will pause thread execution until a fiber has returned. Thejoin non-wait does not pause thread execution but rather provides asuccess/failure status. For both variants, if the instruction isexecuted with no outstanding fiber returns then an exception isgenerated.

The arguments from the returning fiber (up to four) are written toregisters a0-a3.

EFJ

-   -   EFJ.NW        The format for these fiber join instructions is shown Table 7.

TABLE 7 31   25 24  20 19  15 14  12 11  7 6   0 0101000 00000 00000 00000000 1110010 EFJ 0110000 00000 00000 000 00000 1110010 EFJ.NW

Fiber Join All Instructions:

The Fiber Join All instruction (EFJ.ALL) pends until all outstandingfibers have returned. The instruction can be called with zero or morepending fiber returns. No instruction status or exceptions aregenerated. Any returning arguments from the fiber returns are ignored.

-   -   EFJ.ALL        The format for these fiber join all instructions is shown Table        8.

TABLE 8 31   25 24  20 19  15 14  12 11  7 6   0 0111000 00000 00000 00000000 1110010 EFJ.ALL

Atomic Return Instructions:

The EMD atomic return instruction (EAR) is used to complete theexecuting thread of a custom atomic operation and possibly provide aresponse back to the source that issued the custom atomic request.

The EAR instruction can send zero, one, or two 8-byte arguments valueback to the issuing compute element. The number of arguments to sendback is determine by the ac2 suffix (A1 or A2). No suffix means zeroarguments, A1 implies a single 8-byte argument, and A2 implies two8-byte arguments. The arguments, if needed, are obtained from Xregisters a1 and a2.

The EAR instruction is also able to clear the memory line lockassociated with the atomic instruction. The EAR uses the value in the a0register as the address to send the clear lock operation. The clear lockoperation is issued if the instruction contains the suffix CL.

The following DCAS example sends a success or failure back to therequesting processor using the EAR instruction.

// a0 - atomic address // a1 - 64-bit memory value of a0 // a2 - DCAScompare value 1 // a3 - DCAS compare value 2 // a4 - DCAS swap value 1// a5 - DCAS swap value 2 atomic_dcas: bne a1, a2, fail // first 8-bytecompare ld.nb a6, 8(a0) // load second 8-byte memory value - should hitmemory cache bne a6, a3, fail // second 8-byte compare sd a4, 0(a0) //store first 8-byte swap value to thread store buffer sd.cl a5, 8(a0) //store second 8-byte value and clear memory lock li a1, 0 ear.a1 // AMOsuccess response fail: li a1, 1 ear.cl.a1 // AMO failure response (andclear memory lock)

The instruction has two variants that allow the EFT instruction to alsoclear the memory lock associated with the atomic operation. The formatfor the supported instructions is shown in Table 9.

TABLE 9 31   25 24  20 19  15 14  12 11  7 6   0 1010000 00000 00000 ac200000 1110010 EAR 1011000 00000 00000 ac2 00000 1110010 EAR.CL

ac2 Encoding Suffix Argument Count 0 No suffix 0 1 .A1 1 2 .A2 2

First and Second Priority Instructions:

The second (or low) priority instruction transitions the current threadhaving a first priority to a second, low priority. The instruction isgenerally used when a thread is polling on an event to occur (i.e.barrier).

-   -   ELP        The format for the ELP instruction is shown Table 10.

TABLE 10 31   25 24  20 19  15 14  12 11  7 6   0 1000000 00000 00000000 00000 1110010 ELP

The first (or high) priority instruction transitions the current threadhaving a second (or low) priority to a first (or high or normal)priority. The instruction is generally used when a thread is polling andan event has occurred (i.e. barrier).

-   -   ENP        The format for the ENP instruction is shown Table 11.

TABLE 11 31   25 24  20 19  15 14  12 11  7 6   0 1001000 00000 00000000 00000 1110010 ENP

Floating Point Atomic Memory Operations:

Floating point atomic memory operations are performed by the HTP 300associated with a memory controller 120. The floating point operationsperformed are MIN, MAX and ADD, for both 32 and 64-bit data types.

The aq and rl bits in the instruction specify whether all write data isto be visible to other threads prior to issuing the atomic operation(aq), and whether all previously written data should be visible to thisthread after the atomic completes (rl). Put another way, the aq bitforces all write buffers to be written back to memory, and the rl bitforces all read buffers to be invalidated. It should be noted that rs1is an X register value, whereas rd and rs2 are F register values.

-   -   AMOFADD.S rd, rs2, (rs1)    -   AMOFMIN.S rd, rs2, (rs1)    -   AMOFMAX.S rd, rs2, (rs1)    -   AMOFADD.D rd, rs2, (rs1)    -   AMOFMIN.D rd, rs2, (rs1)    -   AMOFMAX.D rd, rs2, (rs1)        The format for these floating point atomic memory operation        instructions is shown Table 12.

TABLE 12 31  27 26 25 24 20 19 15 14 12 11 7 6   0 00000 aq rl rs2 rs1010 rd 0101110 AMOFADD.S 00001 aq rl rs2 rs1 010 rd 0101110 AMOFMIN.S00010 aq rl rs2 rs1 010 rd 0101110 AMOFMAX.S

Custom Atomic Memory Operations:

Custom atomic operations are performed by the HTP 300 associated with amemory controller 120. The operation is performed by executing RISC-Vinstructions. Up to 32 custom atomic operations can be available withinthe memory controllers 120 of a system 100. The custom atomics are asystem wide resource, available to any process attached to the system100.

The aq and rl bits in the instruction specify whether all write data isto be visible to other threads prior to issuing the atomic operation(rl), and whether all previously written data should be visible to thisthread after the atomic completes (aq). Put another way, the rl bitforces all write buffers to be written back to memory, and the aq bitforces all read buffers to be invalidated.

The custom atomics use the a0 register to specify the memory address.The number of source arguments is provided by the suffix (A0, A1, A2 orA4), and are obtained from registers a1-a4. The number of result valuesreturned from memory can be 0-2, and is defined by the custom memoryoperation. The result values are written to register a0-a1.

-   -   AMOCUST0.A4        The following custom atomic instructions are defined as shown in        Table 13.

TABLE 13 31   27 26 25 24  20 19  15 14  12 11  7 6   0 10000 aq rl00000 00000 ac 00000 0101110 AMOCUST0 10001 aq rl 00000 00000 ac 000000101110 AMOCUST1 10010 aq rl 00000 00000 ac 00000 0101110 AMOCUST2 10011aq rl 00000 00000 ac 00000 0101110 AMOCUST3 10100 aq rl 00000 00000 ac00000 0101110 AMOCUST4 10101 aq rl 00000 00000 ac 00000 0101110 AMOCUST510110 aq rl 00000 00000 ac 00000 0101110 AMOCUST6 10111 aq rl 0000000000 ac 00000 0101110 AMOCUST7The ac field is used to specify the number of arguments (0, 1, 2, or 4).The following Table 14 shows the encodings.

TABLE 14 Argument ac Encoding Suffix Count 0 No suffix 0 1 .A1 1 2 .A2 23 .A4 4There are eight custom atomic instructions defined, with 4 argumentcount variants each, resulting a total of 32 possible custom atomicoperators.

Event Management:

The system 100 is an event driven architecture. Each thread has a set ofevents that is able to monitor, utilizing the event received maskregisters 342 and the event state registers 344. Event 0 is reserved fora return from a created fiber (HTP 300 or HTF 200). The remainder of theevents are available for event signaling, either thread-to-thread,broadcast, or collection. Thread-to-thread allows a thread to send anevent to one specific destination thread on the same or a differentnode. Broadcast allows a thread to send a named event to a subset ofthreads on its node. The receiving thread should specify which namedbroadcast event it is expecting. Collection refers to the ability tospecify the number of events that are to be received prior to the eventbecoming active.

An event triggered bit can be cleared (using the EEC instruction), andall events can be listened for (using the EEL instruction). The listenoperation can either pause the thread until an event has triggered, orin non-waiting mode (NW) allowing a thread to periodically poll whileother execution proceeds.

A thread is able to send an event to a specific thread using the eventsend instruction (EES), or broadcast an event to all threads within anode using the event broadcast instruction (EEB). Broadcasted events arenamed events where the sending thread specifies the event name (a 16-bitidentifier), and the receiving threads filter received broadcast eventsfor a pre-specified event identifier. Once received, the event should beexplicitly cleared (EEC) to avoid receiving the same event again. Itshould be noted that all event triggered bits are clear when a threadstarts execution.

Event Mode Instructions:

The event mode (EEM) instruction sets the operation mode for an event.Event 0 is reserved for thread return events, the remainder of theevents can be in one of three receive modes: simple, broadcast, orcollection.

In simple mode, a received event immediately causes the triggered bit tobe set and increments the received message count by one. Each newlyreceived event causes the received event count to be incremented. Thereceive event instruction (EER) causes the received event count to bedecremented by one. The event triggered bit is cleared when the counttransitions back to zero.

In broadcast mode, a received event's channel is compared to the eventnumber's broadcast channel. If the channels match, then the eventtriggered bit is set. The EER instruction causes the triggered bit to becleared.

In collection mode, a received event causes the event trigger count tobe decremented by one. When the count reaches zero, then the eventtriggered bit is set. The EER instruction causes the triggered bit to becleared.

The EEM instruction prepares the event number for the chosen mode ofoperation. In simple mode, the 16-bit event counter is set to zero. Forbroadcast mode, the 16-bit event channel number is set to the valuespecified by the EEM instruction. For collection mode, the 16-bit eventcounter is set to the value specified by the EEM instruction. Each ofthe three modes use the same 16-bit value differently.

-   -   EEM.BM rs1, rs2; rs1=event #, rs2=broadcast channel    -   EEM. CM rs1, rs2; rs1=event #, rs2=collection count    -   EEM. SM rs1; rs1=event #        The format for the event mode instruction is shown Table 15.

TABLE 15 31   25 24  20 19  15 14  12 11  7 6   0 0000100 rs2 rs1 00000000 1110010 EEM.BM 0001100 rs2 rs1 000 00000 1110010 EEM.CM 001010000000 rs1 000 00000 1110010 EEM.SM

Event Destination Instruction:

The event destination (EED) instruction provides an identifier for anevent within the executing thread. The identifier is unique across allexecuting threads within a node. The identifier can be used with theevent send instruction to send an event to the thread using the EESinstruction. The identifier is an opaque value that contains theinformation needed to send the event from a source thread to a specificdestination thread.

The identifier can also be used to obtain a unique value for sending abroadcast event. The identifier includes space for an event number. Theinput register rs1 specifies the event number to encode within thedestination thread identifier. The output rd register contains theidentifier after the instruction executes.

-   -   EED rd, rs1        The format for the event destination instruction is shown Table        16.

TABLE 16 31   25 24  20 19  15 14  12 11  7 6   0 0011100 00000 rs1 000rd 1110010 EED

The event destination instruction can also be utilized by a process toobtain its own address, which can then be used in other broadcastmessages, for example, to enable that process to receive other eventmessages as a destination, e.g., for receiving return messages when theprocess is a master thread.

Event Send Instructions:

The event send (EES) instruction sends an event to a specific thread.Register rs1 provides the destination thread and event number. Registerrs2 provides the optional 8-byte event data.

-   -   EES rs1    -   EES.A1 rs1, rs2        The rs2 register provides the target HTP 300 for the event send        operation. Register rs1 provides the event number to be sent.        Legal values for rs1 are 2-7. The format for the event send        instruction is shown Table 17.

TABLE 17 31   25 24  20 19  15 14  12 11  7 6   0 0100100 00000 rs1 00000000 1110010 EES 0101100 rs2 rs1 000 00000 1110010 EES.A1

Event Broadcast Instructions:

The event broadcast (EEB) instruction broadcasts an event to all threadswithin the node. Register rs1 provides the broadcast channel to be sent(0-65535). Register rs2 provides optional 8-byte event data.

-   -   EEB rs1    -   EEB.A1 rs1, rs2        The format for the event broadcast instruction is shown Table        18.

TABLE 18 31   25 24  20 19  15 14  12 11  7 6   0 0110100 00000 rs1 00000000 1110010 EEB 0111100 rs2 rs1 000 00000 1110010 EEB.A1

Event Listen Instructions:

The event listen (EEL) instruction allows a thread to monitor the statusof received events. The instruction can operate in one of two modes:waiting and non-waiting. The waiting mode will pause the thread until anevent is received, the non-waiting mode provides the received events atthe time the instruction is executed.

-   -   EEL rd, rs1    -   EEL.NW rd, rs1        Register rs1 provides a mask of available events as the output        of the listen operation. The non-waiting mode will return a        value of zero in rs1 if no events are available. The format for        the event listen instructions is shown Table 19.

TABLE 19 31   25 24  20 19  15 14  12 11  7 6   0 1000100 00000 rs1 000rd 1110010 EEL 1001100 00000 rs1 000 rd 1110010 EEL.NW

Event Receive Instructions:

The event receive (EER) instruction is used to receive an event.Receiving an event includes acknowledging that an event was observed,and receiving the optional 8-byte event data. Register rs1 provides theevent number. Register rd contains optional 8-byte event data.

-   -   EER rs1    -   EER.A1 rd, rs1

The format for the event receive instructions is shown Table 20.

TABLE 20 31   25 24  20 19  15 14  12 11  7 6   0 1010100 00000 rs1 00000000 1110010 EER 1011100 00000 rs1 000 rd 1110010 EER.A1

FIG. 10 is a detailed block diagram of a representative embodiment of athread selection control circuitry 405 of the control logic and threadselection circuitry 330 of the HTP 300. As mentioned above, a second orlow priority queue 360 is provided, and thread IDs are selected from thefirst (or high) priority queue 355 or the second or low priority queue360 using a thread selection multiplexer 385, under the control of thethread selection control circuitry 405. Threads in the second priorityqueue 360 are pulled from the queue and executed at a lower rate thanthreads in the first priority queue 360.

As mentioned above, a pair of instructions, ENP and ELP, are used totransition a thread from a first priority to second priority (ELP) andthe second priority to the first priority (ENP).

Threads in a parallel application often must wait for other threads tocomplete priority to resuming execution (i.e., a barrier operation). Thewait operation is completed through communication between the threads.This communication can be supported by an event that wakes a pausedthread, or by the waiting thread polling on a memory location. When athread is polling, it is wasting processing resources that could be usedby the thread that must finish its work to allow all threads to resumeproductive execution. The second or low priority queue 360 allows thewaiting threads to enter a low priority mode that will reduce theoverhead of the polling threads. This serves to reduce the threadexecution overhead of polling threads such that threads that mustcomplete productive work consume the majority of the availableprocessing resources.

A configuration register is used to determine the number of highpriority threads that are to be run for each low priority thread,illustrated in FIG. 10 as the low priority “skip” count, provided to thethread selection control circuitry 405, which selects a thread from thesecond priority queue 360 at predetermined intervals. Stated anotherway, the thread selection multiplexer 385 will select, in succession, apredetermined number (i.e., the skip count) of threads from the firstpriority queue 355, “skipping” selection of any threads from the secondor low priority queue 360. Once that predetermined number of threadsfrom the first priority queue 355 have been selected, the threadselection multiplexer 385 will then select a thread for execution fromthe second priority queue 360, i.e., a predetermined number of highpriority threads are run for each low priority thread. As illustrated,thread selection control circuitry 405 decrements the skip count(register 442, multiplexer 444, and adder 446) until it is equal to zero(logic block 448), at which point the selection input of the threadselection multiplexer 385 toggles to select a thread from the second orlow priority queue 360.

Accordingly, threads in the second priority queue 360 are pulled fromthe queue and executed at a lower rate than threads in the firstpriority queue 355. A configuration register (e.g., in thread memory320) is used to determine the number of high priority threads that areto be run for each low priority thread. A pair of instructions, ENP andELP, are used to transition a thread from first (or normal) priority tothe second, low priority (ELP) and from the second, low priority to thefirst, normal priority (ENP).

FIG. 11 is a diagram of a representative embodiment of a portion of thefirst interconnection network 150 and representative data packets. Inrepresentative embodiment, the first interconnection network 150includes a network bus structure 152 (a plurality of wires or lines), inwhich a first plurality of the network lines 154 are dedicated foraddressing (or routing) data packets (158), and are utilized for settingthe data path through the various crossbar switches, and the remainingsecond plurality of the network lines 156 are dedicated for transmissionof data packets (the data load, illustrated as a train or sequence of“N” data packets 162 ₁ through 162 _(N)) containing operand data,arguments, results, etc.) over the path established through theaddressing lines (first plurality of the network lines 154). Two suchnetwork bus structures 152 are typically provided, into and out of eachcompute resource, as channels, a first channel for receiving data, and asecond channel for transmitting data. A single, first addressing (orrouting) data packet (illustrated as addressing (or routing) data packet158 ₁) may be utilized to establish the routing to a first designateddestination, and may be followed (generally several clock cycles later,to allow for the setting of the switches) by one or more data packets162 which are to be transmitted to the first designated destination, upto a predetermined number of data packets 162 (e.g., up to N datapackets). While that predetermined number of data packets 162 are beingrouted, another, second addressing (or routing) data packet (illustratedas addressing (or routing) data packet 1582) may be transmitted andutilized to establish a routing to a second designated destination, forother, subsequent one or more data packets 162 which will be going tothis second designated destination (illustrated as data packet 162_(N+1)).

FIG. 12 is a detailed block diagram of a representative embodiment ofdata path control circuitry 395 of an HTP 300. As mentioned above, oneor more of the HTPs 300 may also include data path control circuitry395, which is utilized to control access sizes (e.g., memory 125 loadrequests) over the first interconnection network 150 to manage potentialcongestion, providing adaptive bandwidth.

Application performance is often limited by the bandwidth available to aprocessor from memory. The performance limitation can be mitigated byensuring that only data that is needed by an application is brought intothe HTP 300. The data path control circuitry 395 automatically (i.e.,without user intervention) reduces the size of requests to main memory125 to reduce the utilization of the processor interface and memory 125subsystem.

As mentioned above, the compute resources of the system 100 may havemany applications using sparse data sets, with frequent accesses tosmall pieces of data distributed throughout the data set. As a result,if a considerable amount of data is accessed, much of it may be unused,wasting bandwidth. For example, a cache line may be 64 bytes, but notall of it will be utilized. At other times, it will be beneficial to useall available bandwidth, such as for efficient power usage. The datapath control circuitry 395 provides for dynamically adaptive bandwidthover the first interconnection network 150, adjusting the size of thedata path load to optimize performance of any given application, such asadjusting the data path load down to 8-32 bytes (as examples) based uponthe utilization of the receiving (e.g., response) channel of the firstinterconnection network 150 back to the HTP 300.

The data path control circuitry 395 monitors the utilization level onthe first interconnection network 150 and reduces the size of memory 125load (i.e., read) requests from the network interface circuitry 335 asthe utilization increases. In a representative embodiment, the data pathcontrol circuitry 395 performs a time-averaged weighting (time averagedutilization block 364) of the utilization level of the response channelof the first interconnection network 150. If after a fixed period oftime (adjustment interval timer 362) the utilization is above athreshold (and the load request size is greater than the minimum), usingthreshold logic circuit 366 (having a plurality of comparators 482 andselection multiplexers 484, 486), then the size of load requests isreduced by the load request access size logic circuit 368 (generally bya power of 2 (e.g., 8 bytes) from the threshold logic circuit 366, usingminus increment 492), such that: either (a) fewer data packets 162 willbe included in the train of data packets 162, allowing that bandwidth tobe utilized for routing of data packets to another location or foranother process; or (b) memory 125 utilization is more efficient (e.g.,64 bytes are not requested when only 16 bytes will be utilized). Ifafter the fixed period of time the utilization is below a threshold (andthe load request size is less than the maximum), using threshold logiccircuit 366, then the size of the load request is increased by the loadrequest access size logic circuit 368, generally also by a power of 2(e.g., 8 bytes), using plus increment 488. The minimum and maximumvalues for the size of a load request can be user configured, however,the minimum size generally is the size of the issuing load instruction(e.g., the maximum operand size of the HTP 300, such as 8 bytes) and themaximum size is the cache line size (e.g., 32 or 64 bytes). In analternative embodiment, the data path control circuitry 395 can belocated at the memory controller 120, adapting to the bandwidth pressurefrom multiple HTPs 300.

FIG. 13 is a detailed block diagram of a representative embodiment ofsystem call circuitry 415 of an HTP 300 and host interface circuitry115. Representative system 100 embodiments allows a user mode onlycompute element, such as an HTP 300, to perform system calls,breakpoints and other privileged operations without running an operatingsystem, such as to open a file, print, etc. To do so, any of thesesystem operations are originated by an HTP 300 executing a user modeinstruction. The processor's instruction execution identifies that theprocessor must forward the request to a host processor 110 forexecution. The system request from the HTP 300 has the form of systemcall work descriptor packet sent to a host processor 110, and inresponse, the HTP 300 can receive system call return work descriptorpackets.

The system call work descriptor packet, assembled and transmitted by thepacket encoder 380, includes a system call identifier (e.g., a threadID, the core 305 number, a virtual address indicated by the programcounter, the system call arguments or parameters (which are typicallystored in the general purpose registers 328), and return information.The packet is sent to a host interface 115 (SRAM FIFOs 464) that writesto and queues the system call work descriptor packets in a main memoryqueue, such as the illustrated DRAM FIFO 466 in host processor 110 mainmemory, increments a write pointer, and the host interface 115 furtherthen sends an interrupt to the host processor 110 for the host processor110 to poll for a system call work descriptor packet in memory. The hostprocessor's operating system accesses the queue (DRAM FIFO 466) entries,performs the requested operation and places return work descriptor datain a main memory queue (DRAM FIFO 468), and also may signal the hostinterface 115. The host interface 115 monitors the state of the returnqueue (DRAM FIFO 468) and when an entry exists, moves the data into anoutput queue (SRAM output queue 472) and formats a return workdescriptor packet with the work descriptor data provided and sends thereturn work descriptor packet to the HTP 300 which originated the systemcall packet.

The packet decoder 375 of the HTP 300 receives the return workdescriptor packet and places the returned arguments in the generalpurpose registers 328 as if the local processor (HTP 300) performed theoperation itself. This transparent execution as viewed by theapplication running on the user mode HTP 300 results in the ability touse the same programming environment and runtime libraries that are usedwhen a processor has a local operating system, and is highly useful fora wide variety of situations, such as program debugging, using aninserted break point.

The host interface 115, however, typically has limited FIFO space, whichcould be problematic when multiple HTPs 300 are utilized, each having alarge number of cores (e.g., 96), each of which may be running a largenumber of threads (e.g., 32/core). To avoid adding significant memory tothe host interface 115, the overall number of system calls which can besubmitted is limited, using a system call credit mechanism for each HTP300 and each processor core 305 within an HTP 300.

Each processor core 305 includes a first register 452, as part of thesystem call circuitry 415, which maintains a first credit count. Thesystem call circuitry 415, provided per HTP 300, includes a secondregister 458, which includes a second credit count, as a pool ofavailable credits. When a system call work descriptor packet isgenerated, if there are sufficient credits available in the firstregister 452, the system call work descriptor packet may be selected(multiplexer 454) and transmitted, and if not, the system call workdescriptor packet is queued in the system call work descriptor (systemcall) packet table 462, potentially with other system call workdescriptor packet from other processor cores 305 of the given HTP 300.If there are sufficient credits available in the second register 458,providing an extra pool of credits for bursting of system calls andshared among all of the processor cores 305 of the HTP 300, the nextsystem call work descriptor packet may be transmitted, and otherwise isheld in the table.

As those system call work descriptor packets are processed by the hostinterface 115 and read out of the FIFO 464, the host interface 115generates an acknowledgement back to the system call circuitry 415,which increments the credit counts per core in registers 456, which canin turn increment the first credit count in the first register 452, foreach processor core 305.

Alternatively, registers 456 may be utilized equivalently to a firstregister 452, without requiring the separate first register 452 percore, and instead maintaining the first count in the registers 456,again per core 305. As another alternative, all of the system call workdescriptor packets may be queued in the system call work descriptorpacket table 462, on a per core 305 basis, and transmitted when thatcore has sufficient first credit counts in its corresponding register456 or sufficient credits available in the second register 458.

A mechanism is also provided for thread state monitoring, to collect thestate of the set of threads running on an HTP 300 in hardware, whichallows a programmer to have the visibility into the workings of anapplication. With this feature, a host processor 110 can periodicallyaccess and store the information for later use in generating userprofiling reports, for example. With the provided visibility, aprogrammer can make changes to the application to improve itsperformance.

All thread state changes can be monitored and statistics kept on theamount of time in each state. The processor (110 or 300) that iscollecting the statistics provides a means for a separate, secondprocessor (110 or 300) to access and store the data. The data iscollected as the application is running such that a report can beprovided to an application analyst that shows the amount of time in eachstate reported on a periodic basis, which provides detailed visibilityon a running application for later use by an application analyst.

In accordance with the representative embodiments, which may beimplemented in hardware or software, all of the information pertainingto a thread is stored in the various registers of the thread memory 320,and can be copied and saved in another location on a regular basis. Acounter can be utilized to capture the amount of time any given threadspends in a selected state, e.g., a paused state. For example, the hostprocessor 110 can log or capture the current state of all threads andthread counters (amount of time spent in a state), or the differences(delta) between states and counts over time, and write it to a file orotherwise save it in a memory. Also for example, a program or thread maybe a barrier, in which all threads have to complete before anything elsecan start, and it is helpful to monitor which threads are in what stateas they proceed through various barriers or as they change state. Theillustrated code (below) is an example of simulator code which wouldexecute as hardware or be translatable to hardware:

  InStateCount[N]-6b InStateTimeStamp[N]-64b InStateTotalTime[N]-64b

enum ESimR5State { eR5Idle=0, eR5Low=1, eR5Normal=2, eR5PausedMem=3,eR5PausedEar=4,  eR5PausedEel=5, eR5PausedEer=6, eR5PausedEtr=7,eR5PausedEfj=8,  eR5PausedEfjAll=9, eR5PausedSys=10, eR5PausedEes=11 };// set state and collect statistics void setR5CtxState(SimR5HwCtx *pR5Ctx, SimR5HwCtx::ESimR5State state) { m_coreStats.m_coreInStateTotalTime[pR5Ctx->m_r5State] +=   (getSimTime() − m_coreStats.m_coreInStateTime[pR5Ctx->m_r5State]) *  m_coreStats.m_coreInStateCount[pR5Ctx->m_r5State]; m_coreStats.m_coreInStateTime[pR5Ctx->m_r5State] = getSimTime ( ); m_coreStats.m_coreInStateTotalTime[state] +=   (getSimTime ( ) −m_coreStats.m_coreInstateTime[state]) *  m_coreStats.m_coreInStateCount[state]; m_coreStats.m_coreInStateTime[state] = getSimTime ( ); m_coreStats.m_coreInStateCount[pR5Ctx->m_r5State] −= 1; m_coreStats.m_coreInStateCount[state] += 1;  pR5Ctx->m_r5State = state;} void incrementalStateStats(double incStateStats[HTP_R5_STATE_CNT]) { for (int state = 0; state < HTP_R5_STATE_CNT; state += 1) {  m_coreStats.m_coreInStateTotalTime[state] +=    (getSimTime ( ) −m_coreStats.m_coreInStateTime[state]) *   m_coreStats.m_coreInStateCount[state];  m_coreStats.m_coreInStateTime[state] = getSimTime ( );  incStateStats[state] += m_coreStats.m_coreInStateTotalTime[state] −   m_coreStats.m_coreInStatePrevTime[state];  m_coreStats.m_coreInStatePrevTime[state] =m_coreStats.m_coreInStateTotalTime[state];  } }

Numerous advantages of the representative embodiments are readilyapparent. The representative apparatus, system and methods provide for acomputing architecture capable of providing high performance and energyefficient solutions for compute-intensive kernels, such as forcomputation of Fast Fourier Transforms (FFTs) and finite impulseresponse (FIR) filters used in sensing, communication, and analyticapplications, such as synthetic aperture radar, 5G base stations, andgraph analytic applications such as graph clustering using spectraltechniques, machine learning, 5G networking algorithms, and largestencil codes, for example and without limitation.

As used herein, a “processor core” may be any type of processor core,and may be embodied as one or more processor cores configured, designed,programmed or otherwise adapted to perform the functionality discussedherein. As used herein, a “processor” 110 may be any type of processor,and may be embodied as one or more processors configured, designed,programmed or otherwise adapted to perform the functionality discussedherein. As the term processor is used herein, a processor 110 or 300 mayinclude use of a single integrated circuit (“IC”), or may include use ofa plurality of integrated circuits or other components connected,arranged or grouped together, such as controllers, microprocessors,digital signal processors (“DSPs”), array processors, graphics or imageprocessors, parallel processors, multiple core processors, custom ICs,application specific integrated circuits (“ASICs”), field programmablegate arrays (“FPGAs”), adaptive computing ICs, associated memory (suchas RAM, DRAM and ROM), and other ICs and components, whether analog ordigital. As a consequence, as used herein, the term processor orcontroller should be understood to equivalently mean and include asingle IC, or arrangement of custom ICs, ASICs, processors,microprocessors, controllers, FPGAs, adaptive computing ICs, or someother grouping of integrated circuits which perform the functionsdiscussed herein, with associated memory, such as microprocessor memoryor additional RAM, DRAM, SDRAM, SRAM, MRAM, ROM, FLASH, EPROM or E²PROM.A processor 110 or 300, with associated memory, may be adapted orconfigured (via programming, FPGA interconnection, or hard-wiring) toperform the methodology of the invention, as discussed herein. Forexample, the methodology may be programmed and stored, in a processor300 with its associated memory (and/or memory 125) and other equivalentcomponents, as a set of program instructions or other code (orequivalent configuration or other program) for subsequent execution whenthe processor 110 or 300 is operative (i.e., powered on andfunctioning). Equivalently, when the processor 300 may be implemented inwhole or part as FPGAs, custom ICs and/or ASICs, the FPGAs, custom ICsor ASICs also may be designed, configured and/or hard-wired to implementthe methodology of the invention. For example, the processor 110 or 300may be implemented as an arrangement of analog and/or digital circuits,controllers, microprocessors, DSPs and/or ASICs, collectively referredto as a “processor” or “controller”, which are respectively hard-wired,programmed, designed, adapted or configured to implement the methodologyof the invention, including possibly in conjunction with a memory 125.

The memory 125, which may include a data repository (or database), maybe embodied in any number of forms, including within any computer orother machine-readable data storage medium, memory device or otherstorage or communication device for storage or communication ofinformation, currently known or which becomes available in the future,including, but not limited to, a memory integrated circuit (“IC”), ormemory portion of an integrated circuit (such as the resident memorywithin a processor or processor IC), whether volatile or non-volatile,whether removable or non-removable, including without limitation RAM,FLASH, DRAM, SDRAM, SRAM, MRAM, FeRAM, ROM, EPROM or E²PROM, or anyother form of memory device, such as a magnetic hard drive, an opticaldrive, a magnetic disk or tape drive, a hard disk drive, othermachine-readable storage or memory media such as a floppy disk, a CDROM,a CD-RW, digital versatile disk (DVD) or other optical memory, or anyother type of memory, storage medium, or data storage apparatus orcircuit, which is known or which becomes known, depending upon theselected embodiment. The memory 125 may be adapted to store various lookup tables, parameters, coefficients, other information and data,programs or instructions (of the software of the present invention), andother types of tables such as database tables.

As indicated above, the processor 110 or 300 is hard-wired orprogrammed, using software and data structures of the invention, forexample, to perform the methodology of the present invention. As aconsequence, the system and related methods of the present invention,including the various instructions, may be embodied as software whichprovides such programming or other instructions, such as a set ofinstructions and/or metadata embodied within a non-transitory computerreadable medium, discussed above. In addition, metadata may also beutilized to define the various data structures of a look up table or adatabase. Such software may be in the form of source or object code, byway of example and without limitation. Source code further may becompiled into some form of instructions or object code (includingassembly language instructions or configuration information). Thesoftware, source code or metadata of the present invention may beembodied as any type of code, such as C, C++, Matlab, SystemC, LISA,XML, Java, Brew, SQL and its variations (e.g., SQL 99 or proprietaryversions of SQL), DB2, Oracle, or any other type of programming languagewhich performs the functionality discussed herein, including varioushardware definition or hardware modeling languages (e.g., Verilog, VHDL,RTL) and resulting database files (e.g., GDSII). As a consequence, a“construct”, “program construct”, “software construct” or “software”, asused equivalently herein, means and refers to any programming language,of any kind, with any syntax or signatures, which provides or can beinterpreted to provide the associated functionality or methodologyspecified (when instantiated or loaded into a processor or computer andexecuted, including the processor 300, for example).

The software, metadata, or other source code of the present inventionand any resulting bit file (object code, database, or look up table) maybe embodied within any tangible, non-transitory storage medium, such asany of the computer or other machine-readable data storage media, ascomputer-readable instructions, data structures, program modules orother data, such as discussed above with respect to the memory 125,e.g., a floppy disk, a CDROM, a CD-RW, a DVD, a magnetic hard drive, anoptical drive, or any other type of data storage apparatus or medium, asmentioned above.

The present disclosure is to be considered as an exemplification of theprinciples of the invention and is not intended to limit the inventionto the specific embodiments illustrated. In this respect, it is to beunderstood that the invention is not limited in its application to thedetails of construction and to the arrangements of components set forthabove and below, illustrated in the drawings, or as described in theexamples. Systems, methods and apparatuses consistent with the presentinvention are capable of other embodiments and of being practiced andcarried out in various ways.

Although the invention has been described with respect to specificembodiments thereof, these embodiments are merely illustrative and notrestrictive of the invention. In the description herein, numerousspecific details are provided, such as examples of electroniccomponents, electronic and structural connections, materials, andstructural variations, to provide a thorough understanding ofembodiments of the present invention. One skilled in the relevant artwill recognize, however, that an embodiment of the invention can bepracticed without one or more of the specific details, or with otherapparatus, systems, assemblies, components, materials, parts, etc. Inother instances, well-known structures, materials, or operations are notspecifically shown or described in detail to avoid obscuring aspects ofembodiments of the present invention. In addition, the various Figuresare not drawn to scale and should not be regarded as limiting.

Reference throughout this specification to “one embodiment”, “anembodiment”, or a specific “embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention and notnecessarily in all embodiments, and further, are not necessarilyreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics of any specific embodiment of the presentinvention may be combined in any suitable manner and in any suitablecombination with one or more other embodiments, including the use ofselected features without corresponding use of other features. Inaddition, many modifications may be made to adapt a particularapplication, situation or material to the essential scope and spirit ofthe present invention. It is to be understood that other variations andmodifications of the embodiments of the present invention described andillustrated herein are possible in light of the teachings herein and areto be considered part of the spirit and scope of the present invention.

For the recitation of numeric ranges herein, each intervening numberthere between with the same degree of precision is explicitlycontemplated. For example, for the range of 6-9, the numbers 7 and 8 arecontemplated in addition to 6 and 9, and for the range 6.0-7.0, thenumber 6.0, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, and 7.0 areexplicitly contemplated. In addition, every intervening sub-range withinrange is contemplated, in any combination, and is within the scope ofthe disclosure. For example, for the range of 5-10, the sub-ranges 5-6,5-7, 5-8, 5-9, 6-7, 6-8, 6-9, 6-10, 7-8, 7-9, 7-10, 8-9, 8-10, and 9-10are contemplated and within the scope of the disclosed range.

It will also be appreciated that one or more of the elements depicted inthe Figures can also be implemented in a more separate or integratedmanner, or even removed or rendered inoperable in certain cases, as maybe useful in accordance with a particular application. Integrally formedcombinations of components are also within the scope of the invention,particularly for embodiments in which a separation or combination ofdiscrete components is unclear or indiscernible. In addition, use of theterm “coupled” herein, including in its various forms such as “coupling”or “couplable”, means and includes any direct or indirect electrical,structural or magnetic coupling, connection or attachment, or adaptationor capability for such a direct or indirect electrical, structural ormagnetic coupling, connection or attachment, including integrally formedcomponents and components which are coupled via or through anothercomponent.

With respect to signals, we refer herein to parameters that “represent”a given metric or are “representative” of a given metric, where a metricis a measure of a state of at least part of the regulator or its inputsor outputs. A parameter is considered to represent a metric if it isrelated to the metric directly enough that regulating the parameter willsatisfactorily regulate the metric. A parameter may be considered to bean acceptable representation of a metric if it represents a multiple orfraction of the metric.

Furthermore, any signal arrows in the drawings/Figures should beconsidered only exemplary, and not limiting, unless otherwisespecifically noted. Combinations of components of steps will also beconsidered within the scope of the present invention, particularly wherethe ability to separate or combine is unclear or foreseeable. Thedisjunctive term “or”, as used herein and throughout the claims thatfollow, is generally intended to mean “and/or”, having both conjunctiveand disjunctive meanings (and is not confined to an “exclusive or”meaning), unless otherwise indicated. As used in the description hereinand throughout the claims that follow, “a”, “an”, and “the” includeplural references unless the context clearly dictates otherwise. Also asused in the description herein and throughout the claims that follow,the meaning of “in” includes “in” and “on” unless the context clearlydictates otherwise.

The foregoing description of illustrated embodiments of the presentinvention, including what is described in the summary or in theabstract, is not intended to be exhaustive or to limit the invention tothe precise forms disclosed herein. From the foregoing, it will beobserved that numerous variations, modifications and substitutions areintended and may be effected without departing from the spirit and scopeof the novel concept of the invention. It is to be understood that nolimitation with respect to the specific methods and apparatusillustrated herein is intended or should be inferred. It is, of course,intended to cover by the appended claims all such modifications as fallwithin the scope of the claims.

It is claimed:
 1. An apparatus comprising: a processor core configuredto execute a plurality of instructions; and a core control circuitcoupled to the processor core, the core control circuit comprising: aninterconnection network interface configured to transmit one or moresystem call messages and to receive a work descriptor data packet, theinterconnection network interface further configured to decode thereceived work descriptor data packet into an initial program count and areceived argument for a corresponding execution thread; a thread controlmemory comprising a thread identifier register configured to store aplurality of thread identifiers and a program count register configuredto store the initial program count; an execution queue coupled to thethread control memory; a control logic and thread selection circuitcoupled to the execution queue, the control logic and thread selectioncircuit configured, in response to the received work descriptor datapacket, to assign a thread identifier of the plurality of threadidentifiers to the corresponding execution thread, to place the threadidentifier in the execution queue, and to select the thread identifierfor execution by the processor core of an instruction of thecorresponding execution thread, of the plurality of instructions; andsystem call circuitry configured to generate the one or more system callmessages and to modulate a number of the one or more system callmessages transmitted in a predetermined period of time.
 2. The apparatusof claim 1, wherein the interconnection network interface is coupleableto an interconnection network in a system having a host processor, thehost processor configured to receive the one or more system callmessages, to process the one or more system call messages to generateone or more corresponding results, and to generate and transmit a returnwork descriptor packet having the one or more corresponding results tothe interconnection network interface.
 3. The apparatus of claim 2,wherein the interconnection network interface is further configured toreceive and decode the return work descriptor packet having the one ormore corresponding results of the one or more system call messages andto store the one or more corresponding results in the thread controlmemory.
 4. The apparatus of claim 1, wherein the system call circuitrycomprises: one or more system call credit registers configured to storea credit count; and and wherein the system call circuitry is furtherconfigured to queue the one or more system call messages fortransmission when a credit is available from the one or more system callcredit registers.
 5. The apparatus of claim 1, wherein the system callcircuitry comprises: a first system call credit register configured tostore a first credit count; a second system call credit registerconfigured to store a second credit count as a pool of availablecredits; and a system call packet table configured to queue the one ormore system call messages for transmission when a credit is availablefrom the first credit count or the second credit count to modulate thenumber of the one or more system call messages transmitted in thepredetermined period of time.
 6. The apparatus of claim 1, wherein thecontrol logic and thread selection circuit is further configured toautomatically and repeatedly select each thread identifier in theexecution queue for single instruction execution by the processor coreof each corresponding execution thread.
 7. The apparatus of claim 6,wherein the control logic and thread selection circuit is furtherconfigured to assign a valid status or a pause status to the threadidentifier, and for as long as the valid status remains, to return thethread identifier to the execution queue for continued singleinstruction execution by the processor core, and for as long as thepause status remains, to pause thread execution by not returning thethread identifier to the execution queue.
 8. The apparatus of claim 7,wherein the control logic and thread selection circuit is furtherconfigured to assign the pause status to the thread identifier when theprocessor core has executed a memory load instruction or a memory storeinstruction for the corresponding execution thread.
 9. The apparatus ofclaim 7, wherein the control logic and thread selection circuit isfurther configured to change a status of the thread identifier from thepause status to the valid status in response to a received event datapacket to resume execution of the corresponding execution thread or inresponse to an event number of the received event data packet to resumeexecution of the corresponding execution thread.
 10. The apparatus ofclaim 1, wherein the interconnection network interface is furtherconfigured to generate a return work descriptor packet in response to anexecution of a return instruction by the processor core.
 11. Theapparatus of claim 1, wherein the interconnection network interface isfurther configured to receive an event data packet, and to decode thereceived event data packet into an event identifier and any receivedargument.
 12. The apparatus of claim 11, wherein the thread controlmemory further comprises an event mask register configured to store anevent mask and wherein the control logic and thread selection circuit isfurther configured to determine an event number corresponding to thereceived event data packet and to use the event mask to respond to thereceived event data packet.
 13. The apparatus of claim 1, wherein thethread control memory further comprises a data cache configured to storecached data and a general-purpose register configured to store thereceived argument.
 14. The apparatus of claim 1, wherein the processorcore is configured to execute a fiber create instruction, wherein thecore control circuit is further configured to generate one or more workdescriptor data packets for execution of a corresponding plurality ofexecution threads, and wherein the control logic and thread selectioncircuit is further configured to reserve a predetermined amount ofmemory space in the thread control memory for storage of returnarguments.
 15. The apparatus of claim 1, wherein the core controlcircuit further comprises: a network response memory; an instructioncache coupled to the control logic and thread selection circuit; and anetwork command queue configured to store one or more requests or returninformation for transmission by the interconnection network interface onthe interconnection network.
 16. The apparatus of claim 1, wherein thecontrol logic and thread selection circuit is further configured to endexecution of the corresponding execution thread and to return the threadidentifier to the thread identifier pool register in response to anexecution of a return instruction by the processor core for thecorresponding execution thread.
 17. The apparatus of claim 16, whereinthe control logic and thread selection circuit is further configured toclear one or more registers of the thread control memory indexed by thethread identifier of the corresponding execution thread in response tothe execution of the return instruction by the processor core.
 18. Theapparatus of claim 1, wherein the execution queue further comprises: afirst priority queue configured to store one or more thread identifiersfor first priority execution threads; and a second priority queueconfigured to store one or more thread identifiers for second priorityexecution threads; and wherein the control logic and thread selectioncircuit further comprises: thread selection control circuitry coupled tothe execution queue, the thread selection control circuitry configuredto select a thread identifier, of the one or more thread identifiers forfirst priority execution threads from the first priority queue, at afirst frequency for execution by the processor core of an instruction ofa first priority execution thread, and to select a thread identifier, ofthe one or more thread identifiers for second priority execution threadsfrom the second priority queue, at a second frequency for execution bythe processor core of an instruction of a second priority executionthread, the second frequency lower than the first frequency.
 19. Anapparatus comprising: a processor core configured to execute a pluralityof instructions; and a core control circuit coupled to the processorcore, the core control circuit comprising: an interconnection networkinterface configured to transmit one or more system call messages, toreceive a work descriptor data packet, and to decode the received workdescriptor data packet into an initial program count and any receivedargument for a corresponding execution thread; a thread control memorycoupled to the interconnection network interface; an execution queuecoupled to the thread control memory; an instruction cache coupled tothe processor core; a control logic and thread selection circuit coupledto the execution queue, to the instruction cache, and to the threadcontrol memory, the control logic and thread selection circuitconfigured, in response to the received work descriptor data packet, toassign a thread identifier of a plurality of thread identifiers to thecorresponding execution thread, to place the thread identifier in theexecution queue, and to periodically select the thread identifier fromthe execution queue for instruction execution by the processor core ofthe corresponding execution thread; and system call circuitry configuredto generate the one or more system call messages, the system callcircuitry comprising: a first system call credit register configured tostore a first credit count; a second system call credit registerconfigured to store a second credit count as a pool of availablecredits; and a system call packet table configured to queue the one ormore system call messages for transmission when a credit is availablefrom the first credit count or the second credit count to modulate anumber of the one or more system call messages transmitted in apredetermined period of time.
 20. An apparatus comprising: a processorcore configured to execute a plurality of instructions; and a corecontrol circuit coupled to the processor core, the core control circuitcomprising: an interconnection network interface configured to receive acall work descriptor data packet, to decode the received call workdescriptor data packet into an initial program count and a receivedargument for a corresponding execution thread, to encode and transmitone or more system call messages, and to receive and decode a returnwork descriptor packet having one or more corresponding results of theone or more system call messages; a thread control memory coupled to theinterconnection network interface, the thread control memory comprisinga thread identifier pool register configured to store a plurality ofthread identifiers, a program count register configured to store theinitial program count, and a general-purpose register configured tostore the received argument; an execution queue coupled to the threadcontrol memory; a control logic and thread selection circuit coupled tothe execution queue and to the thread control memory, the control logicand thread selection circuit configured, in response to the receivedcall work descriptor data packet, to assign a thread identifier of theplurality of thread identifiers to the corresponding execution thread,to place the thread identifier in the execution queue, and to select thethread identifier from the execution queue for instruction execution bythe processor core of the corresponding execution thread; an instructioncache coupled to the control logic and thread selection circuit, theinstruction cache configured to receive the initial program count andprovide to the processor core a corresponding instruction for execution,of the plurality of instructions; and system call circuitry configuredto generate the one or more system call messages, the system callcircuitry comprising one or more system call credit registers configuredto store a credit count, the system call circuitry further configured toqueue the one or more system call messages for transmission when acredit is available from credit count to modulate a number of the one ormore system call messages transmitted to a host processor in apredetermined period of time.